User Guide#

Overview#

tlmnet provides exact optimization for Quantized Statistical Learning with Ternary Linear Models (TLMs). The core estimator, TlmMilpClassifier, optimizes linear models whose feature weights are restricted to discrete ternary states:

\[w_j \in \{-1, 0, 1\}\]

Instead of relying on greedy heuristic approximations or continuous \(L_1\) shrinkage penalties, tlmnet uses Mixed-Integer Linear Programming (MILP) via scipy.optimize.milp to identify globally optimal weight vectors.

Mathematical Background#

Decision Variable Splitting#

To express non-convex ternary weight constraints within a linear programming framework, each weight \(w_j\) is decomposed into two binary indicator variables:

\[w_j = u_j - v_j, \quad \text{where } u_j, v_j \in \{0, 1\}\]

To prevent simultaneous activation (\(u_j=1\) and \(v_j=1\)), the model enforces mutual exclusivity:

\[u_j + v_j \le 1 \quad \forall j \in \{1, \dots, p\}\]

Soft-Margin MILP Formulation#

Given a dataset \((X, y)\) with binary targets \(y_i \in \{-1, +1\}\), TlmMilpClassifier minimizes total classification slack penalties \(\sum_{i=1}^n \xi_i\):

\[\min_{u, v, \xi} C \sum_{i=1}^n \xi_i\]

subject to the classification margin constraints:

\[y_i \left( \sum_{j=1}^p X_{ij} (u_j - v_j) \right) + \xi_i \ge 1, \quad \xi_i \ge 0\]

Exact L0 Feature Budgeting#

An exact upper bound on feature selection cardinality can be imposed via the max_features parameter, enforcing a strict \(L_0\) constraint:

\[\sum_{j=1}^p (u_j + v_j) \le k\]

Pipeline Patterns & Preprocessing#

Feature Standardization#

Because feature values directly multiply discrete weights \(w_j \in \{-1, 0, 1\}\), feature scaling heavily influences margin penalties. Dense numeric features should be standardized to zero mean and unit variance using StandardScaler.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from tlmnet import TlmMilpClassifier

clf = Pipeline([
    ('scaler', StandardScaler()),
    ('tlm', TlmMilpClassifier(max_features=5, C=1.0))
])

Sparse Data Support#

TlmMilpClassifier natively processes sparse data representations. By leveraging scipy.sparse block matrix assembly, it constructs the constraint equations directly from CSR, CSC, or COO matrices without ever converting the dataset to a dense array. This makes it highly efficient for high-dimensional text classification pipelines using TfidfVectorizer.

Hyperparameter Configuration#

  • ``max_features`` (int or None, default=None): Maximum number of non-zero ternary weights allowed in the model (\(L_0\) cardinality constraint). If None, feature inclusion is unconstrained.

  • ``C`` (float, default=1.0): Penalty parameter for classification margin slacks. Higher values penalize misclassifications more heavily.

  • ``time_limit`` (float, default=60.0): Maximum allowable time in seconds allocated to the underlying MILP branch-and-bound solver. If the solver reaches this limit before proving global optimality, it automatically recovers and returns the best incumbent integer solution found.

  • ``mip_rel_gap`` (float, default=1e-4): Relative MIP gap tolerance for early branch-and-bound termination. Allows the solver to stop once the incumbent solution is within this percentage of the theoretical optimal bound.

  • ``verbose`` (bool, default=False): Enables real-time solver output logging during optimization to monitor branch-and-bound progress.

Handling Multiclass Targets#

TlmMilpClassifier strictly supports binary classification targets. To use ternary linear models on multiclass problems, wrap the estimator in Scikit-Learn’s OneVsRestClassifier or OneVsOneClassifier.

from sklearn.multiclass import OneVsRestClassifier
from tlmnet import TlmMilpClassifier

ovr_clf = OneVsRestClassifier(TlmMilpClassifier(max_features=10, C=1.0))
ovr_clf.fit(X_train, y_train)