.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/plot_cumulative_auc_by_feature.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_plot_cumulative_auc_by_feature.py: ==================================================================== Cumulative AUC by Feature: Forward Selection Trajectory ==================================================================== This example visualizes the internal greedy forward-selection mechanics of ``CalfCV``. At each step, ``CalfCV`` evaluates all unselected features and appends the single feature (and coarse ±1 weight direction) that yields the highest cumulative ROC-AUC sum. The process terminates automatically when the AUC gain falls below ``auc_tol``. The weights (+1, -1) assigned at each step are displayed directly above the data points, while the corresponding feature added at each step is labeled along the X-axis. .. GENERATED FROM PYTHON SOURCE LINES 20-22 Imports and Model Fitting ------------------------- .. GENERATED FROM PYTHON SOURCE LINES 22-46 .. code-block:: Python import matplotlib.pyplot as plt import numpy as np import pandas as pd from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from calfcv import CalfCV X, y = load_breast_cancer(return_X_y=True, as_frame=True) feature_names = X.columns.values X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, random_state=42, stratify=y ) clf = CalfCV( grid=[(-1, 1), (-1, 0, 1)], auc_tol=[1e-4, 1e-3, 1e-2], order_col=[True, False], cv=5, n_jobs=-1, ) clf.fit(X_train, y_train) .. raw:: html 
CalfCV(auc_tol=[0.0001, 0.001, 0.01], cv=5, grid=[(-1, 1), (-1, 0, 1)],
           n_jobs=-1, order_col=[True, False])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.


.. GENERATED FROM PYTHON SOURCE LINES 47-49 Extract Forward-Selection Steps ------------------------------- .. GENERATED FROM PYTHON SOURCE LINES 49-64 .. code-block:: Python best_calf = clf.model_.best_estimator_["classifier"] # Because of the patch to `_utils.py`, these three arrays are now # guaranteed to be the exact same length. cumulative_aucs = best_calf.auc_ selected_indices = best_calf.feature_index_ assigned_weights = best_calf.weights_ steps = np.arange(1, len(cumulative_aucs) + 1) # Format X-axis tick labels as "1. Feature Name" x_labels = [ f"{step}. {feature_names[idx]}" for step, idx in zip(steps, selected_indices) ] .. GENERATED FROM PYTHON SOURCE LINES 65-67 Plot Clean Trajectory with Minimal Point Annotations ---------------------------------------------------- .. GENERATED FROM PYTHON SOURCE LINES 67-125 .. code-block:: Python fig, ax = plt.subplots(figsize=(10, 6)) # Plot cumulative AUC curve ax.plot( steps, cumulative_aucs, marker="o", markersize=8, color="#1f77b4", linewidth=2.5, label="Cumulative Training ROC-AUC", ) # Minimal point annotations: Just display '+1' or '-1' directly above vertex for step, auc, weight in zip(steps, cumulative_aucs, assigned_weights): weight_str = f"+{int(weight)}" if weight > 0 else f"{int(weight)}" ax.annotate( weight_str, (step, auc), xytext=(0, 10), textcoords="offset points", ha="center", va="bottom", fontsize=10, fontweight="bold", color="#1f77b4" if weight > 0 else "#d62728", ) # Highlight early-stopping cutoff line optimal_tol = clf.best_params_["classifier__auc_tol"] ax.axhline( y=cumulative_aucs[-1], color="gray", linestyle="--", linewidth=1.5, label=f"Early Stopping Plateau (auc_tol={optimal_tol})", ) # Configure crisp X-axis tick labels ax.set_xticks(steps) ax.set_xticklabels(x_labels, rotation=35, ha="right", fontsize=9) ax.set_ylabel("Cumulative ROC-AUC", fontsize=11) ax.set_title( "CALF Forward Selection: Cumulative AUC by Feature Step", fontsize=12, fontweight="bold", ) ax.grid(True, linestyle="--", alpha=0.5) ax.legend(loc="lower right") # Padding for point labels ymin, ymax = ax.get_ylim() ax.set_ylim(ymin, ymax + 0.02) plt.tight_layout() plt.show() .. image-sg:: /auto_examples/images/sphx_glr_plot_cumulative_auc_by_feature_001.png :alt: CALF Forward Selection: Cumulative AUC by Feature Step :srcset: /auto_examples/images/sphx_glr_plot_cumulative_auc_by_feature_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 126-128 Step-by-Step Selection Summary ------------------------------ .. GENERATED FROM PYTHON SOURCE LINES 128-141 .. code-block:: Python summary_df = pd.DataFrame( { "Step": steps, "Feature Added": [feature_names[i] for i in selected_indices], "Coarse Weight": [f"{int(w):+d}" for w in assigned_weights], "Cumulative AUC": np.round(cumulative_aucs, 4), } ) print("\n" + "=" * 55) print("GREEDY FEATURE SELECTION STEP SUMMARY") print("=" * 55) print(summary_df.to_string(index=False)) .. rst-class:: sphx-glr-script-out .. code-block:: none ======================================================= GREEDY FEATURE SELECTION STEP SUMMARY ======================================================= Step Feature Added Coarse Weight Cumulative AUC 1 mean radius -1 0.9370 2 mean perimeter -1 0.9422 3 mean area +1 0.9445 4 mean smoothness -1 0.9454 5 mean concavity -1 0.9532 6 mean concave points -1 0.9588 7 mean fractal dimension +1 0.9755 8 area error -1 0.9775 9 concave points error +1 0.9782 10 worst radius -1 0.9809 11 worst texture -1 0.9904 12 worst smoothness -1 0.9915 13 worst compactness -1 0.9917 .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 3.455 seconds) .. _sphx_glr_download_auto_examples_plot_cumulative_auc_by_feature.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_cumulative_auc_by_feature.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_cumulative_auc_by_feature.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_cumulative_auc_by_feature.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_