.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/plot_decision_thresholds.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_plot_decision_thresholds.py: ======================================================================== Decision Threshold Calibration: Precision, Recall, and F1 Trade-offs ======================================================================== When a classifier generates continuous decision scores or probabilities, changing the decision threshold (the cutoff point where a score becomes a hard 0 or 1 classification) shifts classification metrics. Moving the threshold up catches fewer positives (higher precision, lower recall), while moving the threshold down catches more positives (lower precision, higher recall). This example illustrates how Precision, Recall, and F1-Score dynamically trade off across decision thresholds ranging from 0.0 to 1.0 for a trained :class:`CalfCV` estimator. .. GENERATED FROM PYTHON SOURCE LINES 18-19 Imports and Synthetic Dataset Generation .. GENERATED FROM PYTHON SOURCE LINES 19-42 .. code-block:: Python import matplotlib.pyplot as plt import numpy as np from sklearn.datasets import make_classification from sklearn.metrics import precision_recall_curve from sklearn.model_selection import train_test_split from calfcv import CalfCV # Generate synthetic dataset scaled for rapid local/CI builds (N=250, P=50) X, y = make_classification( n_samples=250, n_features=50, n_informative=20, n_redundant=10, n_classes=2, shuffle=False, random_state=42, ) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, random_state=42, stratify=y ) .. GENERATED FROM PYTHON SOURCE LINES 43-44 Fit CalfCV Estimator via Automated Grid Search .. GENERATED FROM PYTHON SOURCE LINES 44-53 .. code-block:: Python clf = CalfCV( grid=[(-1, 1), (-1, 0, 1)], auc_tol=[1e-3, 1e-2], order_col=[True, False], cv=3, n_jobs=-1, ) clf.fit(X_train, y_train) .. raw:: html 
CalfCV(auc_tol=[0.001, 0.01], cv=3, grid=[(-1, 1), (-1, 0, 1)], n_jobs=-1,
           order_col=[True, False])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.


.. GENERATED FROM PYTHON SOURCE LINES 54-57 Compute Probabilities and Precision-Recall Curve CalfCV internally applies a sigmoid transformation to its integer-weighted sum decision scores to provide continuous probability estimates between 0 and 1. .. GENERATED FROM PYTHON SOURCE LINES 57-77 .. code-block:: Python probabilities = clf.predict_proba(X_test)[:, 1] precision, recall, thresholds = precision_recall_curve(y_test, probabilities) # Calculate F1-Score across all thresholds f1_scores = 2 * (precision[:-1] * recall[:-1]) / (precision[:-1] + recall[:-1] + 1e-10) # Identify threshold maximizing F1-Score best_idx = np.argmax(f1_scores) best_threshold = thresholds[best_idx] best_f1 = f1_scores[best_idx] print("\n" + "=" * 50) print("THRESHOLD CALIBRATION METRICS") print("=" * 50) print(f"Optimal F1 Threshold: {best_threshold:.4f}") print(f"Peak F1-Score: {best_f1:.4f}") print(f"Precision at Peak: {precision[best_idx]:.4f}") print(f"Recall at Peak: {recall[best_idx]:.4f}") .. rst-class:: sphx-glr-script-out .. code-block:: none ================================================== THRESHOLD CALIBRATION METRICS ================================================== Optimal F1 Threshold: 0.5757 Peak F1-Score: 0.8267 Precision at Peak: 0.8158 Recall at Peak: 0.8378 .. GENERATED FROM PYTHON SOURCE LINES 78-79 Plot Precision, Recall, and F1 Trade-offs Across Thresholds .. GENERATED FROM PYTHON SOURCE LINES 79-121 .. code-block:: Python plt.figure(figsize=(9, 5.5)) plt.plot(thresholds, precision[:-1], label="Precision", color="#1f77b4", linewidth=2) plt.plot(thresholds, recall[:-1], label="Recall", color="#2ca02c", linewidth=2) plt.plot( thresholds, f1_scores, label="F1-Score", color="#d62728", linestyle="--", linewidth=2, ) # Highlight peak F1 threshold point plt.axvline( x=best_threshold, color="gray", linestyle=":", linewidth=1.5, label=f"Max F1 Threshold ({best_threshold:.2f})", ) plt.scatter( [best_threshold], [best_f1], color="#d62728", s=80, zorder=5, ) plt.xlabel("Decision Threshold (Probability Estimate)", fontsize=11) plt.ylabel("Metric Score", fontsize=11) plt.title( "CalfCV Decision Threshold Calibration (Precision vs. Recall vs. F1)", fontsize=12, fontweight="bold", ) plt.legend(loc="lower left", fontsize=10) plt.grid(True, linestyle="--", alpha=0.5) plt.xlim(0.0, 1.0) plt.ylim(0.0, 1.05) plt.tight_layout() plt.show() .. image-sg:: /auto_examples/images/sphx_glr_plot_decision_thresholds_001.png :alt: CalfCV Decision Threshold Calibration (Precision vs. Recall vs. F1) :srcset: /auto_examples/images/sphx_glr_plot_decision_thresholds_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 122-139 Threshold Calibration Analysis ------------------------------ 1. Native Probability Estimates: `CalfCV` natively computes `predict_proba` by mapping its coarse integer-weighted sums through a sigmoid function. This provides calibrated probabilities suitable for continuous threshold tuning. 2. Precision-Recall Trade-off: In diagnostic contexts where false positives are costly, raising the threshold yields higher precision. In screening contexts where catching all positives is critical, lowering the threshold prioritizes recall. 3. Optimal Threshold Boundary (Vertical Dotted Line): The vertical dotted line highlights the exact cutoff that maximizes the F1-Score (the harmonic mean of precision and recall). This visually demonstrates whether the decision boundary needs to be tuned higher or lower than the default 0.50 cutoff for optimal metric balance on hold-out data. .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 5.433 seconds) .. _sphx_glr_download_auto_examples_plot_decision_thresholds.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_decision_thresholds.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_decision_thresholds.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_decision_thresholds.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_