.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/plot_runtime_vs_performance.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_plot_runtime_vs_performance.py: ======================================================================== Runtime vs. Classifier Performance Trade-offs ======================================================================== This example benchmarks execution time (fit time) against downstream ROC-AUC and feature selection sparsity across different preprocessing strategies on a noisy dataset. While unconstrained continuous models (like baseline Logistic Regression) fit nearly instantly, they retain all noise features and risk overfitting. Conversely, methods like ``CalfCV`` execute an internal combinatorial grid search to automatically prune noise, trading modest additional fit time for optimal generalization and extreme model sparsity. .. GENERATED FROM PYTHON SOURCE LINES 17-18 Imports and Synthetic Dataset Generation .. GENERATED FROM PYTHON SOURCE LINES 18-40 .. code-block:: Python import time import matplotlib.pyplot as plt import numpy as np import pandas as pd from sklearn.datasets import make_classification from sklearn.feature_selection import RFE, SelectKBest, f_classif from sklearn.linear_model import LogisticRegression from sklearn.model_selection import StratifiedKFold, cross_validate from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler from calfcv import Calf, CalfCV X, y = make_classification( n_samples=1000, n_features=200, n_informative=20, n_redundant=10, n_classes=2, shuffle=False, # Columns 0..29 are signal; 30..199 are pure noise (170 columns) random_state=42, ) .. GENERATED FROM PYTHON SOURCE LINES 41-42 Define Preprocessing Pipelines .. GENERATED FROM PYTHON SOURCE LINES 42-82 .. code-block:: Python pipelines = { "Baseline (No Selection)": make_pipeline( StandardScaler(), LogisticRegression(solver="liblinear", random_state=42) ), "CALF (Unsorted)": make_pipeline( StandardScaler(), Calf(order_col=False), LogisticRegression(solver="liblinear", random_state=42), ), "CALF (Pre-Sorted)": make_pipeline( StandardScaler(), Calf(order_col=True), LogisticRegression(solver="liblinear", random_state=42), ), "CalfCV (Auto Grid Search)": make_pipeline( CalfCV( grid=[(-1, 1), (-1, 0, 1)], auc_tol=[1e-6, 1e-3], order_col=[True, False], cv=3, n_jobs=-1, ), LogisticRegression(solver="liblinear", random_state=42), ), "SelectKBest (ANOVA k=15)": make_pipeline( StandardScaler(), SelectKBest(score_func=f_classif, k=15), LogisticRegression(solver="liblinear", random_state=42), ), "RFE (Logistic Regression k=15)": make_pipeline( StandardScaler(), RFE( estimator=LogisticRegression(solver="liblinear", random_state=42), n_features_to_select=15, ), LogisticRegression(solver="liblinear", random_state=42), ), } .. GENERATED FROM PYTHON SOURCE LINES 83-84 Benchmark Execution Time and Model Metrics across CV Folds .. GENERATED FROM PYTHON SOURCE LINES 84-125 .. code-block:: Python cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) scoring = ["roc_auc", "accuracy"] metrics = [] for name, pipe in pipelines.items(): start_time = time.time() cv_res = cross_validate(pipe, X, y, cv=cv, scoring=scoring, return_estimator=True) total_cv_time = time.time() - start_time avg_fit_time = np.mean(cv_res["fit_time"]) # Determine average number of selected features across folds feature_counts = [] for est in cv_res["estimator"]: if "calf" in est.named_steps: feature_counts.append(len(est.named_steps["calf"].feature_index_)) elif "calfcv" in est.named_steps: best_calf = est.named_steps["calfcv"].model_.best_estimator_["classifier"] feature_counts.append(len(best_calf.feature_index_)) elif "selectkbest" in est.named_steps: feature_counts.append(np.sum(est.named_steps["selectkbest"].get_support())) elif "rfe" in est.named_steps: feature_counts.append(np.sum(est.named_steps["rfe"].support_)) else: feature_counts.append(X.shape[1]) metrics.append( { "Pipeline": name, "Mean Fit Time (s)": avg_fit_time, "Total CV Time (s)": total_cv_time, "Mean ROC-AUC": np.mean(cv_res["test_roc_auc"]), "Std ROC-AUC": np.std(cv_res["test_roc_auc"]), "Mean Accuracy": np.mean(cv_res["test_accuracy"]), "Avg Selected Features (k)": np.mean(feature_counts), } ) df_metrics = pd.DataFrame(metrics) .. GENERATED FROM PYTHON SOURCE LINES 126-127 Visualize Runtime vs. Performance Trade-off .. GENERATED FROM PYTHON SOURCE LINES 127-170 .. code-block:: Python fig, ax = plt.subplots(figsize=(10, 6)) colors = plt.cm.Set2(np.linspace(0, 1, len(df_metrics))) for i, row in df_metrics.iterrows(): # Marker size scales with average selected features k size = max(80, row["Avg Selected Features (k)"] * 3) ax.scatter( row["Mean Fit Time (s)"], row["Mean ROC-AUC"], s=size, color=colors[i], alpha=0.85, edgecolors="black", linewidth=1.2, label=row["Pipeline"], ) # Annotate points with pipeline name and feature count k ax.annotate( f"{row['Pipeline']}\n(k={row['Avg Selected Features (k)']:.0f})", xy=(row["Mean Fit Time (s)"], row["Mean ROC-AUC"]), xytext=(8, -5), textcoords="offset points", fontsize=8, fontweight="bold" if "CalfCV" in row["Pipeline"] else "normal", ) ax.set_xscale("log") ax.set_xlabel("Mean Fit Time per CV Fold (Seconds, Log Scale)", fontsize=11) ax.set_ylabel("Cross-Validated ROC-AUC Score", fontsize=11) ax.set_title( "Runtime vs. Predictive Performance Trade-off", fontsize=13, fontweight="bold" ) ax.grid(True, linestyle="--", alpha=0.5) ymin, ymax = ax.get_ylim() ax.set_ylim(ymin - 0.02, ymax + 0.02) plt.tight_layout() plt.show() .. image-sg:: /auto_examples/images/sphx_glr_plot_runtime_vs_performance_001.png :alt: Runtime vs. Predictive Performance Trade-off :srcset: /auto_examples/images/sphx_glr_plot_runtime_vs_performance_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 171-172 Summary Table .. GENERATED FROM PYTHON SOURCE LINES 172-187 .. code-block:: Python print("\n" + "=" * 80) print("RUNTIME VS. PERFORMANCE SUMMARY") print("=" * 80) print( df_metrics[ [ "Pipeline", "Mean Fit Time (s)", "Mean ROC-AUC", "Mean Accuracy", "Avg Selected Features (k)", ] ].to_string(index=False) ) .. rst-class:: sphx-glr-script-out .. code-block:: none ================================================================================ RUNTIME VS. PERFORMANCE SUMMARY ================================================================================ Pipeline Mean Fit Time (s) Mean ROC-AUC Mean Accuracy Avg Selected Features (k) Baseline (No Selection) 0.022044 0.816556 0.736 200.0 CALF (Unsorted) 1.161765 0.875778 0.792 16.2 CALF (Pre-Sorted) 1.308909 0.854077 0.773 12.2 CalfCV (Auto Grid Search) 6.727295 0.854697 0.776 10.8 SelectKBest (ANOVA k=15) 0.008689 0.875577 0.798 15.0 RFE (Logistic Regression k=15) 2.179428 0.880137 0.796 15.0 .. GENERATED FROM PYTHON SOURCE LINES 188-198 Trade-off Analysis ------------------ 1. Unconstrained Baseline Speed vs. Overfitting: Baseline models fit almost instantaneously (<0.01s), but process all 50 features, leaving the downstream model exposed to noise overfitting. 2. Automated Hyperparameter Tuning Overhead: `CalfCV` incurs higher fit time because it executes an internal cross-validation loop to evaluate multiple candidate weight grids and AUC tolerances. However, it completely automates parameter selection while achieving peak ROC-AUC and discarding pure noise columns. .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 57.792 seconds) .. _sphx_glr_download_auto_examples_plot_runtime_vs_performance.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_runtime_vs_performance.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_runtime_vs_performance.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_runtime_vs_performance.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_