Twelve models were benchmarked on 29.9 million LendingClub records to answer a narrower question than "will this loan default": will this application even be graded at all — the first real checkpoint before any question of default risk. The champion model scored a suspicious-looking 1.000 across every metric. This case study doesn't hide that number or wave it away — it explains exactly where it comes from.
The dataset is a combined LendingClub accepted-and-rejected loan file, 29,907,694 records — LendingClub's public "accepted" applications (the ones that went through full underwriting and received a credit grade) merged with its public "rejected" applications (auto-declined, never graded). Rather than jumping straight to predicting default risk, the project first asks a simpler, earlier question: will an application receive a grade at all, formulated as binary classification — No Grade versus Graded. Twelve classification models were benchmarked on it: Decision Tree, Random Forest, Extra Trees, Logistic Regression, Naive Bayes, KNN, SGD, Ridge Classifier, LightGBM, AdaBoost, CatBoost, and XGBoost.
An early pass benchmarked these models with the human-assigned credit Grade included as a feature — and that version leaked in an obvious way: Grade is only ever assigned to applications that get graded, so a model that can see Grade is answering a circular question. That leak was caught and removed. The production feature set — NO_GRADE — was deliberately restricted to three financial fields only: Loan Amount, Annual Income, and DTI (debt-to-income ratio), forcing every model to find real signal in ordinary underwriting parameters instead of the label in disguise.
On the NO_GRADE feature set, Decision Tree came out on top with a perfect 1.000000 across accuracy, precision, recall, F1, and ROC-AUC — training in 10.857 seconds and predicting in 0.157 seconds. Random Forest and LightGBM scored nearly as well but were far more expensive: Random Forest took 432.9 seconds to train, and AdaBoost (LightGBM's boosting sibling in this benchmark) took 2,128.8 seconds. KNN also scored near-perfect but was disqualified from production consideration outright — its prediction time on the full dataset was 2,631 seconds, unusable for anything resembling a live decision.
| Model | Accuracy / ROC-AUC | Train time | Outcome |
|---|---|---|---|
| Decision Tree | 1.000000 | 10.857s | Selected — perfect score, fastest by far |
| Random Forest | ≈1.000 | 432.9s | Near-perfect, too slow to justify over DT |
| AdaBoost | ≈1.000 | 2,128.8s | Near-perfect, far too slow |
| KNN | ≈1.000 | — | Disqualified — 2,631s prediction time on full data |
| Ridge Classifier | 0.999979 AUC | — | Worst — recall 0.117, F1 0.209 despite the AUC |
annual_inc at 0.9999999697,
with loan_amnt and dti essentially at zero: the tree is, in practice, a
single split on "is annual income zero." Checking why: in the raw data, 100% of the 27,648,741
No‑Grade rows have annual_inc == 0.00, versus only 8 of the 2,258,953 Graded rows.
That single fact separates the two classes almost perfectly on its own.
This isn't circular in the way the Grade leak was, because it isn't a coincidence of how the dataset
was assembled — it's a real fact about LendingClub's own process. An application only has its
income verified once it proceeds to full underwriting, which is the same event as being graded.
Whether income was ever collected is information a live system would already have before deciding
whether to grade an application, so the model is using a legitimate, available signal — it's
just an unusually decisive one, and the near-perfect score reflects that rather than a modeling error.
Two separate checks confirmed the result held up outside the benchmark split. First, the champion and the worst model were re-run on the full 29.9-million-row dataset, and the same ranking held at production scale, not just on a held-out sample. Second, a more pointed test: the original, Grade-containing dataset was fed into the Grade-blind NO_GRADE champion model, and it still reproduced historical approve/reject decisions with near-perfect accuracy — proof the model was correctly ignoring a column it was never trained on, not quietly relying on it through some backdoor in the data pipeline.
The champion Decision Tree and the worst-performing Ridge Classifier were built into a side-by-side desktop demo with identical inputs, to make the practical stakes of model choice concrete rather than abstract. On one real test scenario — a $100,000 loan request against $100,000 annual income and a DTI of 1 — the two models disagreed outright: the champion approved it, Ridge rejected it. The same applicant, two different outcomes, depending entirely on which model was making the call. The report frames a human-AI workflow on top of this: routine applications get an instant AI-driven decision, while complex or policy-sensitive cases escalate to a human loan officer rather than being auto-decided.