Project 6 turns ESA's public Collision Avoidance (Kelvins) dataset into a working decision-support tool — from a rebuilt dataset and a disciplined EDA record, through an eight-model benchmark run across four increasingly rigorous passes, to a deployed, human-in-the-loop application that puts the winning model directly in front of the person who has to act on it.
Given a predicted close approach between two objects in orbit, should an operator treat it as actionable — worth reviewing and possibly maneuvering for — or is it safe to leave to routine monitoring? That's the decision this project builds toward: not a black-box verdict, but a transparent, honestly-benchmarked signal a human reviewer can trust and act on.
Source: ESA's Kelvins Collision Avoidance Challenge — 162,634 raw Conjunction Data Message (CDM) rows,
each one a snapshot of a predicted close approach that gets refreshed as tracking data refines. The
dataset was rebuilt to one row per event (the CDM closest to actual closest approach, the most refined
assessment available), giving 13,154 events across 108 columns. The target —
actionable — is a binary flag (risk > −6 on a log₁₀ collision-probability
scale), chosen over regression because 63% of last-CDM rows sit at a reporting floor that's a censoring
artifact, not a real measurement. Positive rate: 2.77% (365 of 13,154 events).
Mahalanobis distance — a covariance-normalized version of miss distance — turned out to be the single strongest discriminator: an 8× gap between actionable events (mean 51) and non-actionable ones (mean 402). Risk also varies sharply by the type of object involved:
Actionable rate by chaser object type. Payload's low rate survives a distance-band control — its full physical mechanism remains an open question, documented rather than papered over.
Following the same disciplined pattern as the Workforce Income Analytics project: every feature-set comparison changes exactly one variable, so the result reflects the feature set alone. Experiment A (Baseline): the full 102-feature raw set. Experiment B (Curated): 14 domain-selected features an operator would actually reason about. Experiment A beat Experiment B on every one of five models, on both ROC-AUC and F1, without exception — removing 88 features cost XGBoost a 26% relative drop in F1. The cause was traceable: the single most important feature in the winning model (target orbital inclination) wasn't in the curated list at all.
| Model | ROC-AUC (A) | ROC-AUC (B) | Best-F1 (A) | Best-F1 (B) |
|---|---|---|---|---|
| XGBoost | 0.9794 | 0.9184 | 0.596 | 0.438 |
| CatBoost | 0.9720 | 0.9135 | 0.574 | 0.430 |
| Random Forest | 0.9725 | 0.9171 | 0.538 | 0.377 |
| Logistic Regression | 0.8763 | 0.8653 | 0.301 | 0.274 |
| SVM | 0.8753 | 0.7454 | 0.301 | 0.182 |
Model selection didn't stop at the first result that looked good — it moved through four distinct passes, each one catching something the last one missed.
Five algorithms, fixed hyperparameters. A correlation-only leakage check had cleared two alternative risk columns as "weakly correlated" — retraining with them included pushed ROC-AUC to a suspicious 1.0000. Both columns were excluded from every model after that.
Re-run with genuine GridSearchCV hyperparameter search rather than fixed defaults. XGBoost and CatBoost emerged as the top two, essentially tied at the Recall≥80% operating point.
Extra Trees, LightGBM, and HistGradientBoosting joined the field. LightGBM landed a genuine podium finish — 2nd of all eight models by both ROC-AUC and Best-F1.
The three strongest models trained with full-scale hyperparameter grids. This pass also caught and fixed a real checkpoint bug that could have silently shipped a stale model as a genuine retrain — see "Engineering rigor" below.
| Rank | Model | Test ROC-AUC | Precision @ 80%+ Recall | Recall |
|---|---|---|---|---|
| 1 | XGBoost (FULL) | 0.9814 | 39.9% | 80.8% |
| 2 | LightGBM (FULL) | 0.9791 | 37.7% | 83.6% |
| 3 | CatBoost (FULL) | 0.9702 | 33.9% | 80.8% |
Evaluated at the Recall≥80% operating point — a deliberate product decision for an advance-warning system, where missing a real actionable event costs more than an extra manual review.
A second, complementary research thread: how does the orbital environment's aggregate risk change as the rocket-body population grows? Scoped deliberately to rocket bodies — spent upper stages, directly countable from launch cadence — and fit to three confirmed real data points.
| Scenario | Growth × | Rocket bodies | Aggregate risk |
|---|---|---|---|
| 2015–19 baseline | 1× | 1,869 | −4.27 |
| Real 2025 launch cadence | 3.39× | ≈6,336 | −3.74 |
| Megaconstellation scenario | 535× | ≈1M | −1.54 |
Worldwide launches climbed from a 95.6/year average (2015–19) to 324 in 2025 — a real, already- happened 3.39× increase, not a projection. At today's real cadence, aggregate risk sits around −3.74: a modest, real shift. The named threshold — −2, "as congested as a Bangkok highway in space" — sits well short of today, but past the real, researched 535× scenario the model is capped at, keeping the tool honest about what's happened versus what's merely possible.
A real, working two-lens application — not a mock-up. A FastAPI backend serves both a live scoring endpoint and the frontend from a single process.
Lens 1 — Bangkok Highway Index (shown first, built for public awareness): an interactive growth-scenario dial letting a non-technical viewer drag from today's real traffic level up to the real, researched worst case and feel the difference — rocket-body count, actionable close calls per year, and a visible "crisis mode" once the named threshold is crossed.
Lens 2 — Live Conjunction Alert (the operational lens): upload a CDM batch as CSV and score it live against the real, trained FULL-mode XGBoost model — verified end-to-end, reproducing the model's own logged probabilities to within floating-point noise. Each row carries a "Your Call" control so a human reviewer can record their own judgment independently of the model, with a live summary of where they agree or differ — the system surfaces the signal, the operator decides.
A decision-support system earns trust not by being flawless on the first attempt, but by catching its own mistakes before they reach the person relying on it.