Nine models were benchmarked to predict customer churn on IBM's Telco dataset. The model with the highest accuracy wasn't the one that went into production — because accuracy wasn't the question that mattered. The question was which model's behavior matched a real business threshold, and that's a different exercise entirely.
The dataset is the standard IBM/Kaggle Telco Customer Churn set — 7,043 customers, 29 fields, a 26.54% churn rate, mean tenure 32.37 months, mean monthly charge $64.76. Nine classification algorithms were benchmarked for the core churn prediction task (Decision Tree, Random Forest, Logistic Regression, LinearSVC, MLP, NuSVC, SVM, XGBoost, CatBoost), alongside two supporting models: an SVR regressor to predict tenure, and a OneClassSVM for anomaly screening.
At the default 0.5 decision threshold, CatBoost posted the best raw accuracy (81.05%), with XGBoost and CatBoost tied for the best ROC-AUC (≈0.854). NuSVC badly underperformed — 40.5% accuracy, ROC-AUC of just 0.169 — and was ruled out immediately. On paper, CatBoost or XGBoost looked like the obvious choice.
| Model | Accuracy | ROC-AUC | Cross-point (months) | Selected? |
|---|---|---|---|---|
| CatBoost | 81.05% | ≈0.854 | — | No — 12.5h train time |
| XGBoost | ≈80% | ≈0.854 | — | No — 23.6h train time |
| SVM | 80.6% | 0.847 | 41 | Yes — closest to the 40-month business threshold |
| NuSVC | 40.5% | 0.169 | — | No — ruled out immediately |
SVM's full metrics: 80.6% accuracy, 66.3% precision, 54.3% recall, 59.7% F1, 84.7% ROC-AUC — and the best best-threshold F1 (0.646) of any model tested, not just the closest cross-point.
There was a practical factor behind the decision too, not just the philosophical one: XGBoost took roughly 23.6 hours to train on this dataset and CatBoost roughly 12.5 hours, against a fraction of that for SVM and LinearSVC. A model that takes a day to retrain is a real operational cost, and it stacked on top of, not instead of, the cross-point reasoning above.
SVR was selected as the tenure-prediction model that pairs with the SVM classifier, and it fit extremely well: R² = 0.9998, MAE of 0.19 months, predicting a best tenure of 55.00079 against an actual value of 55. Together, the two models let the system flag not just whether a customer is likely to churn, but roughly when in their tenure that risk peaks.
Both models were built into a working desktop application — the Telco AI Target Prediction Application (part of the AI-CRDSS decision-support suite) — that loads a customer CSV, classifies churn risk, predicts tenure, filters and ranks customers by risk, and exports results to CSV, Excel, PDF, or Power BI. It's already been run against real data: a production batch on 6 July 2026 flagged 2,720 of 7,043 customers (38.6%) against the 40/55-month thresholds.