Taking a churn model from notebook to production API

Context

The client sells project-management software to mid-sized companies on annual and monthly plans. Like most subscription businesses, they lose a share of customers every year, and their customer success team wanted to reach at-risk accounts before renewal conversations started.

A data scientist on their analytics team had built a churn model in a Jupyter notebook. It used product usage, support tickets and billing history, and in offline tests it ranked at-risk accounts far better than the rule of thumb the team was using. Then the data scientist moved to another role, and the model stayed in the notebook.

The challenge

The model was promising, but nobody could use it:

• No way to call it. The product team wanted a churn score in the admin dashboard and in the CRM. There was no API, only a notebook that had to be run by hand.
• No way to reproduce it. Training depended on a local CSV export, hard-coded dates and a specific version of a library on one laptop.
• No record of what had been trained. Several versions of the model existed, with no way to tell which one had produced the results people had seen.
• No warning if it went wrong. If usage patterns shifted, for example after a pricing change, nothing would flag that the model had become less accurate.

Our approach

Reproduce first, improve second

We resisted the temptation to start tuning. The first milestone was simply to reproduce the notebook's results from code, starting from the source databases rather than the CSV export. That surfaced two subtle issues: a feature that accidentally used information from after the prediction date, and a filter that excluded customers who had already churned. Fixing them lowered the offline score slightly but made it honest.

A training pipeline anyone can run

We restructured the notebook into a small Python package with separate steps for data extraction, feature building, training and evaluation. Each step has tests, and the whole pipeline runs with one command. Every run is logged to MLflow with its parameters, data snapshot, metrics and the resulting model, so any score in production can be traced back to the exact run that produced it.

Serving predictions

The model is wrapped in a FastAPI service with two endpoints: one returns a score for a single account, the other returns scores for a batch. The service loads the currently approved model from the MLflow registry at start-up, validates every request against a schema and returns the score along with the model version that produced it.

We containerised the service and deployed it to the client's existing AWS setup alongside their other services, using their deployment pipeline rather than introducing a new one.

Monitoring

We added lightweight monitoring that the platform team can extend:

• Input checks that flag missing or out-of-range features.
• A weekly job comparing the distribution of incoming features with the training data, to catch drift early.
• A monthly comparison of predicted churn against actual outcomes once renewal dates pass.

How it works

1. On the first of each month, a scheduled job runs the training pipeline against the latest data.
2. The new model is evaluated against the current production model on the same recent period.
3. If it performs at least as well and passes the input checks, it is registered as the new candidate and a short report is posted to the team's channel.
4. An engineer approves the promotion with one command. The API picks up the new version on its next deployment.

Keeping a human approval step was a deliberate choice. Fully automatic promotion is possible, but for a model that influences customer conversations, the client preferred a quick review.

Results

The API serves predictions in under 50 ms at the 95th percentile, fast enough to show scores live in the admin dashboard. Monthly retraining now runs with no manual steps up to the approval point, and every training run is tracked and reproducible.

More importantly, the client's own engineers have retrained and redeployed the model twice since handover without our involvement.

What we delivered

• A tested Python package containing the full training pipeline.
• Experiment tracking and a model registry in MLflow.
• A containerised FastAPI inference service with OpenAPI documentation.
• Monitoring for input quality, feature drift and prediction accuracy.
• Architecture notes, a runbook and a recorded walkthrough for the engineering team.

What we learned

Most ML projects that stall do so between "the model works" and "the model is used". The work here was mostly ordinary software engineering applied carefully to ML: tests, versioning, clear interfaces and documentation. That is what made the model something the client could own.