.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/feature_selection/plot_rfe_with_cross_validation.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_feature_selection_plot_rfe_with_cross_validation.py>`
        to download the full example code. or to run this example in your browser via Binder

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_feature_selection_plot_rfe_with_cross_validation.py:


===================================================
Recursive feature elimination with cross-validation
===================================================

A Recursive Feature Elimination (RFE) example with automatic tuning of the
number of features selected with cross-validation.

.. GENERATED FROM PYTHON SOURCE LINES 10-14

.. code-block:: Python


    # Authors: The scikit-learn developers
    # SPDX-License-Identifier: BSD-3-Clause








.. GENERATED FROM PYTHON SOURCE LINES 15-22

Data generation
---------------

We build a classification task using 3 informative features. The introduction
of 2 additional redundant (i.e. correlated) features has the effect that the
selected features vary depending on the cross-validation fold. The remaining
features are non-informative as they are drawn at random.

.. GENERATED FROM PYTHON SOURCE LINES 22-37

.. code-block:: Python


    from sklearn.datasets import make_classification

    X, y = make_classification(
        n_samples=500,
        n_features=15,
        n_informative=3,
        n_redundant=2,
        n_repeated=0,
        n_classes=8,
        n_clusters_per_class=1,
        class_sep=0.8,
        random_state=0,
    )








.. GENERATED FROM PYTHON SOURCE LINES 38-43

Model training and selection
----------------------------

We create the RFE object and compute the cross-validated scores. The scoring
strategy "accuracy" optimizes the proportion of correctly classified samples.

.. GENERATED FROM PYTHON SOURCE LINES 43-64

.. code-block:: Python


    from sklearn.feature_selection import RFECV
    from sklearn.linear_model import LogisticRegression
    from sklearn.model_selection import StratifiedKFold

    min_features_to_select = 1  # Minimum number of features to consider
    clf = LogisticRegression()
    cv = StratifiedKFold(5)

    rfecv = RFECV(
        estimator=clf,
        step=1,
        cv=cv,
        scoring="accuracy",
        min_features_to_select=min_features_to_select,
        n_jobs=2,
    )
    rfecv.fit(X, y)

    print(f"Optimal number of features: {rfecv.n_features_}")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Optimal number of features: 3




.. GENERATED FROM PYTHON SOURCE LINES 65-70

In the present case, the model with 3 features (which corresponds to the true
generative model) is found to be the most optimal.

Plot number of features VS. cross-validation scores
---------------------------------------------------

.. GENERATED FROM PYTHON SOURCE LINES 70-86

.. code-block:: Python


    import matplotlib.pyplot as plt
    import pandas as pd

    cv_results = pd.DataFrame(rfecv.cv_results_)
    plt.figure()
    plt.xlabel("Number of features selected")
    plt.ylabel("Mean test accuracy")
    plt.errorbar(
        x=cv_results["n_features"],
        y=cv_results["mean_test_score"],
        yerr=cv_results["std_test_score"],
    )
    plt.title("Recursive Feature Elimination \nwith correlated features")
    plt.show()




.. image-sg:: /auto_examples/feature_selection/images/sphx_glr_plot_rfe_with_cross_validation_001.png
   :alt: Recursive Feature Elimination  with correlated features
   :srcset: /auto_examples/feature_selection/images/sphx_glr_plot_rfe_with_cross_validation_001.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 87-94

From the plot above one can further notice a plateau of equivalent scores
(similar mean value and overlapping errorbars) for 3 to 5 selected features.
This is the result of introducing correlated features. Indeed, the optimal
model selected by the RFE can lie within this range, depending on the
cross-validation technique. The test accuracy decreases above 5 selected
features, this is, keeping non-informative features leads to over-fitting and
is therefore detrimental for the statistical performance of the models.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.403 seconds)


.. _sphx_glr_download_auto_examples_feature_selection_plot_rfe_with_cross_validation.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: binder-badge

      .. image:: images/binder_badge_logo.svg
        :target: https://mybinder.org/v2/gh/scikit-learn/scikit-learn/main?urlpath=lab/tree/notebooks/auto_examples/feature_selection/plot_rfe_with_cross_validation.ipynb
        :alt: Launch binder
        :width: 150 px

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_rfe_with_cross_validation.ipynb <plot_rfe_with_cross_validation.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_rfe_with_cross_validation.py <plot_rfe_with_cross_validation.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_rfe_with_cross_validation.zip <plot_rfe_with_cross_validation.zip>`


.. include:: plot_rfe_with_cross_validation.recommendations


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_