Telco Churn Advisory System — Case Study

Nine models, and the obvious winner wasn't chosen

The dataset is the standard IBM/Kaggle Telco Customer Churn set — 7,043 customers, 29 fields, a 26.54% churn rate, mean tenure 32.37 months, mean monthly charge $64.76. Nine classification algorithms were benchmarked for the core churn prediction task (Decision Tree, Random Forest, Logistic Regression, LinearSVC, MLP, NuSVC, SVM, XGBoost, CatBoost), alongside two supporting models: an SVR regressor to predict tenure, and a OneClassSVM for anomaly screening.

At the default 0.5 decision threshold, CatBoost posted the best raw accuracy (81.05%), with XGBoost and CatBoost tied for the best ROC-AUC (≈0.854). NuSVC badly underperformed — 40.5% accuracy, ROC-AUC of just 0.169 — and was ruled out immediately. On paper, CatBoost or XGBoost looked like the obvious choice.

Neither was picked. Instead of taking the leaderboard at face value, the team first went back to the Power BI dashboard and found a real crossover point in the data: satisfaction and churn rate cross at roughly 40 months of tenure — a business-meaningful threshold, not a modeling artifact. Each candidate model has its own internal "cross-point" — the tenure value at which its predicted churn probability crosses the decision boundary. SVM's cross-point landed at 41 months, the closest match to the business threshold of any model tested. SVM was picked for that reason, not because it had the best accuracy.

Model	Accuracy	ROC-AUC	Cross-point (months)	Selected?
CatBoost	81.05%	≈0.854	—	No — 12.5h train time
XGBoost	≈80%	≈0.854	—	No — 23.6h train time
SVM	80.6%	0.847	41	Yes — closest to the 40-month business threshold
NuSVC	40.5%	0.169	—	No — ruled out immediately

SVM's full metrics: 80.6% accuracy, 66.3% precision, 54.3% recall, 59.7% F1, 84.7% ROC-AUC — and the best best-threshold F1 (0.646) of any model tested, not just the closest cross-point.

There was a practical factor behind the decision too, not just the philosophical one: XGBoost took roughly 23.6 hours to train on this dataset and CatBoost roughly 12.5 hours, against a fraction of that for SVM and LinearSVC. A model that takes a day to retrain is a real operational cost, and it stacked on top of, not instead of, the cross-point reasoning above.

A companion regression model

SVR was selected as the tenure-prediction model that pairs with the SVM classifier, and it fit extremely well: R² = 0.9998, MAE of 0.19 months, predicting a best tenure of 55.00079 against an actual value of 55. Together, the two models let the system flag not just whether a customer is likely to churn, but roughly when in their tenure that risk peaks.

From models to a working tool

Both models were built into a working desktop application — the Telco AI Target Prediction Application (part of the AI-CRDSS decision-support suite) — that loads a customer CSV, classifies churn risk, predicts tenure, filters and ranks customers by risk, and exports results to CSV, Excel, PDF, or Power BI. It's already been run against real data: a production batch on 6 July 2026 flagged 2,720 of 7,043 customers (38.6%) against the 40/55-month thresholds.

Update (July 2026): a browser-based version of this tool is now live — try it here — running the same SVM/SVR pipeline described above, no install required. That closes the deployment gap from the original roadmap, but the substance gaps are unchanged: it still runs against a fixed CSV snapshot rather than accepting a general file upload, the role selector is a UI convenience rather than real multi-user authentication, there's no automated retraining, and per-prediction explainability (the planned SHAP layer) hasn't been added yet. A churn score without a reason attached to it is exactly the kind of black-box output this site's whole philosophy pushes against — that piece, along with an audit/database layer and enterprise Power BI analytics integration, is still ahead.