Retention
Flags accounts drifting toward cancellation with the signals behind the flag, assigns an intervention, and holds back a control group so the intervention can actually be measured.
Build effort, not elapsed calendar time. Engagements ran alongside study and employment, so this figure is the work itself rather than the span it sat in.
Interface built in code for this case study — not a client screen capture.
01Business problem
Subscription businesses find out a customer has left when the cancellation arrives, even though the behaviour that predicted it was visible in their own data for weeks.
Who feels it
The signals are usually all present: logins tapering, a champion user who stopped appearing, a support ticket that was resolved but not really, an invoice paid late for the first time, a feature that was central to the original use case going untouched. Each sits in a different system and none of them triggers anything.
The common response is a churn score on a dashboard. This almost always fails for two reasons. The first is that nobody knows what to do with a number, so the dashboard is opened for a month and then not again. The second is more subtle and more damaging: the score is usually built on features that leak the outcome. If "days since last login" is computed at report time — not as of a cutoff sitting before the outcome window — the model learns that customers who have not logged in for ninety days churn. True, useless, and it scores like accuracy.
The third failure is that nobody can tell whether retention efforts worked. A CSM calls at-risk accounts, some renew, and the team concludes the programme is working. Without a control group there is no way to distinguish the intervention from the accounts that were never going to leave.
02Product overview
A retention system built on an explicit churn definition, features computed strictly as of a cutoff date to prevent leakage, risk flags that name their contributing signals, playbook-driven interventions assigned to owners, and a deterministic holdout so intervention effect is measured and never assumed.
Engineering notes
It starts with a definition. Churn is stated precisely — for a contract business, non-renewal within a window after term end; for a self-serve one, cancellation or a defined period of inactivity. Without that, every downstream number is unfalsifiable. The definition is configuration, because it genuinely differs by business model.
Features are computed as of a cutoff. Every signal — engagement trend, breadth of feature use, active seat ratio, support sentiment trend, payment behaviour, champion activity — is calculated using only data available before the cutoff, and the prediction window sits strictly after it. This is unglamorous and it is the difference between a model that predicts and one that describes the past.
Risk is presented with its reasons and, crucially, with an action. Each risk pattern maps to a playbook entry — an owner, a suggested intervention, a timeframe — so what lands in front of a human is a task, not a number. Assignment is recorded, and the outcome of every intervention is tracked against it.
A fraction of eligible at-risk accounts is deterministically held back from intervention. It is uncomfortable and it is the only way to answer whether the programme works. Holdout assignment is a hash of the account id so it is stable across runs, and the interface shows the comparison honestly, including when the difference is not yet distinguishable from noise.
03Key features
10 capabilities, of which 2 are model-backed. The rest are ordinary deterministic software, and the distinction is marked so the AI claim stays honest.
Churn is defined per business model in configuration, with the prediction window stated, so every metric downstream is falsifiable.
Every signal is computed as of a cutoff date using only prior data — the line between a predictive model and a well-dressed description of the past.
Engagement trend, feature breadth, active seat ratio, support sentiment, payment behaviour and champion activity, each visible separately.
A flagged account shows the contributing signals and their direction, not a score in isolation.
The users whose activity historically carries an account are identified, and their disappearance is treated as a distinct signal.
Each risk pattern maps to an owner, a suggested action and a timeframe, so the output is work to be done and not a dashboard to be read.
A deterministic fraction of eligible at-risk accounts is held back from intervention so effect can be measured.
Every intervention records what was done, by whom, and what happened — renewed, downgraded, churned, still open.
Retention by signup cohort, plan and segment, with the churn definition applied consistently across all of them.
Comparisons state when a difference is not yet distinguishable from noise instead of reporting a favourable number.
04User flow
The path from the trigger to the finished record, with each step attributed to whoever performs it — a person or the system.
05Architecture
The processing path first, then the layers it runs on, then the constraints that shaped both.
Path through the system
Churn definition and prediction window set per business model.
Features built as of a cutoff, with prior-data-only enforcement.
Risk assessed with contributing signals retained per account.
Playbook match to an owner and action, or deterministic holdout.
Intervention executed and recorded against the account.
Outcomes compared against holdout with uncertainty reported.
Layers
Why it is shaped this way
06Technical decisions
Every decision here was contested by a reasonable alternative. The trade-off column is the part usually left out.
Why
Target leakage is the default failure of churn modelling. The cutoff is enforced in the computation layer and never left to each query to filter correctly, so leakage is structurally difficult and not merely discouraged.
Trade-off
Snapshots consume storage and the pipeline is more complex than querying current state. I accepted both; a cheaper pipeline that produces an unusable number is not a saving.
Why
Non-renewal after a contract term and thirty days of inactivity in a self-serve product are both churn, and a system that hardcodes one cannot serve the other. Making it explicit also forces the business to state what it actually means.
Trade-off
Metrics are only comparable across periods when the definition has not changed, so the definition is versioned and reports name the version used.
Why
Without a control, intervention effect is unmeasurable and every programme appears to work. A stable hash-based holdout gives a comparison without maintaining group membership tables that drift.
Trade-off
Some at-risk accounts receive no intervention. That is a real cost, bounded to a small fraction, and it buys the only honest answer to whether the rest of the effort is worth spending.
Why
A score changes nobody’s behaviour. A named pattern with an owner and a suggested action does — and when the assessment is wrong, the pattern is the part a human can argue with.
Trade-off
Playbook coverage has to keep pace with observed patterns, and an unmatched pattern needs a default owner so nothing falls through.
07Challenges
Not the setup work. These are the problems where the first implementation was wrong and had to be reconsidered.
Approach
Feature computation was rebuilt around a required cutoff parameter, with the aggregation layer refusing events at or after the cutoff. Labels are drawn strictly from the window after it. Validation splits by time, never at random, so a model is always evaluated on a period it has not seen.
Outcome
Measured performance dropped substantially and became real. That drop was the most useful result in the project.
Approach
Engagement is compared against the account’s own trailing baseline and against a segment-level index for the same period, so a decline that every account shares does not flag individually. Accounts with too little history are marked insufficient-data and left unscored.
Outcome
A December-wide slowdown stopped generating a wave of false flags that would have discredited the whole system.
Approach
The account page leads with the specific signals that moved, each as a small trend against its baseline, followed by recent support interactions and the last intervention. Everything is server-rendered so the page arrives complete, and the suggested action sits at the top of it, not the bottom.
Outcome
The screen became something opened before a call. It stopped being a weekly report that quietly went unread.
08Business value
Stated qualitatively on purpose. Invented percentages are the easiest thing to put on a portfolio and the easiest thing to see through.
Operational effect
What would change at scale
09Interface
Dense operational views rather than dashboards. These are used for hours at a time, so the priorities are legibility, keyboard flow and state that is never ambiguous.
Account health list
Accounts by risk band with the leading signal, owner, renewal date and whether they are in the holdout.
Account detail
The signals that moved, each as a trend against the account’s own baseline, with recent support history and the suggested action first.
Intervention queue
Assigned actions with owner, due date and recorded outcome per account.
Programme effect
Intervention group against holdout, with the comparison stated as inconclusive when it is.
The screen above is built in HTML and CSS for this case study. It reproduces the layout, states and vocabulary of the real build without exposing client data, which is why it exists rather than a screenshot. It is evidence of design and of the decisions behind it. It is not a photograph of a deployed system, and no part of it is a capture of anyone else’s product.
10Technologies
Chosen for the shape of the problem, not for novelty. Anything unusual is justified in the decisions section above.
11Technical preview
There is no repository link on this site. What is available instead is the module structure, the data model, the interface contracts and annotated excerpts written for this case study — enough for a technical reviewer to judge the engineering.
Technical preview
Selected implementation details are available for technical review: component structure, data models, architecture notes and sanitized code excerpts. Some client-specific source material stays private because of confidentiality. Enter the project access keyword if you have been given one — the same one works across every case study.
A presentation convenience, not authentication. This is a static site, so the keyword ships to your browser with the page and anyone reading the bundle can find it — worth saying plainly rather than dressing it up. Nothing confidential is stored behind it.
Inside this section
For private client work, contact me for a walkthrough.
Next step
Send the process, the constraints and the deadline. You will get an honest scope, an architecture sketch and a timeline before any commitment.