Health Score Backtest Playbook 2026

The quarter closing in Salesforce fires the play, and the payload it needs is historical: each account's health score as it stood 90 days before its renewal decision, not the score today. Today's score has already absorbed the churn — dead accounts drift red on the way out and flatter the model.

• Snapshot scores weekly into your warehouse. If you aren't doing this yet, it is the first task of this play — you can't backtest a score you didn't keep.
• Pull renewal outcomes, churn reason codes and expansion events from the CRM in the same extract. Reason codes matter downstream: a score shouldn't be penalized for missing an acquisition-driven churn no behavior predicts.

One SQL join in the warehouse: score-at-T-minus-90 against outcome-at-renewal. Out comes a two-by-two every RevOps operator should know by heart — red or green, churned or retained — and the three numbers this play runs on:

• Recall (the misses): of the accounts that churned, what share were red 90 days out? This is churned-while-green, inverted — the number that decides whether CSMs should believe the score at all.
• Precision (the false alarms): of the accounts flagged red, what share actually churned or contracted? Below ~30%, CSMs learn to ignore red, and the score dies of alert fatigue.
• Lead time: for the churns the score did catch, how many days before the renewal decision did the account first go red? A correct flag 10 days out is trivia; the same flag at 120 days is a save motion.

Field report: the two numbers that matter most are churned-while-green and lead time. A score can post great topline accuracy and still be useless — right about the accounts nobody worried about, late on the ones everybody did.

Did green accounts churn? Set the bar in advance and write it down — a common line is: more than 20% of churned ARR was green at T-minus-90 means the model gets rebuilt, not tuned. If greens held, skip straight to publishing the scorecard; a passing grade is worth showing too, and skipping the audit on good quarters is how scores rot unnoticed.

Now the operator work. Pull the feature weights and interrogate each input against the misses:

• Login counts — the classic green-churn generator. Accounts log in to export their data on the way out. Kill it or cap its weight.
• NPS — a relationship survey answered by your fans, on a lag. It predicts who answers surveys.
• CSM sentiment — it encodes the same optimism the forecast already has; as a score input it just launders gut-feel into a number.

Reweight toward behavioral engagement — breadth, depth and frequency of real product use, active-seat percentage, the features gone quiet. This is exactly the layer Accoil computes from the events you already send to Segment, PostHog, Amplitude or Mixpanel, so the reweight is configuration, not a data engineering project. Change a small number of weights per quarter and note what changed — next quarter's backtest has to be attributable, or you're just stirring.

Post the backtest to the CS team's channel on a schedule, whether the news is good or bad: the confusion matrix, churned-while-green, false-alarm rate, lead time, and what was reweighted and why. Four numbers and one change log — if it doesn't fit in a Slack message, it won't be read.

Publishing in public is the mechanism, not a nicety. CSMs don't distrust health scores because CSMs are stubborn; they distrust scores that have never been audited where they can see it. A score that shows its misses buys the right to be believed on its flags.

The play succeeds when red accounts get a response inside the SLA because the CS team believes red means something. Watch the override rate as your trust gauge: a score nobody overrides is as suspicious as one everyone overrides — the first means nobody's engaging with it, the second means nobody believes it. Healthy is a modest override rate with logged reasons, feeding the next quarter's backtest. That loop — score, outcome, audit, reweight — is the difference between a health score and health score theater.