How to measure renewal-alert quality without inventing saved revenue

A renewal alert can be correct without saving a customer. It can also be commercially useful even when the account eventually leaves. These possibilities make product evaluation harder than counting tasks or attaching the contract value to every flagged record. A useful study needs to separate the quality of detection, the quality of follow-up and the customer's eventual decision. Collapsing those into a single saved-revenue number makes the result easier to market and harder to trust.

This article proposes an internal evaluation method. It does not publish an industry benchmark, a measured Renewal Radar result or a retention forecast. Every sample size and review window should be chosen for the team's portfolio and decision. The method is designed to produce inspectable evidence before anyone makes claims about financial impact.

Name the population and the operating unit

Define the population as the accounts or renewal decisions that were eligible during a stated window. Record the portal, extraction time, date rule, excluded records and relevant product scope. If a company has several renewal events, decide whether the unit is the company, contract or subscription. Mixing those units can inflate the apparent success of a queue because several commercial decisions may be represented by one convenient account label.

Preserve the original queue at the start of review. Include flagged and unflagged eligible records. If the evaluator sees only alerts, they can inspect false positives but cannot discover missed accounts. This is the most important sampling choice in the study. A beautifully accurate list of alerts can still leave the majority of actionable work outside the list.

Create an independent review label

Ask a reviewer to inspect the underlying records and classify whether an actionable follow-up gap existed at the original observation time. Provide written definitions and evidence requirements before the review. Avoid letting the product's own label become the reference answer. Where practical, hide the automated decision until the reviewer has completed their assessment. This reduces the temptation to rationalize the existing classification.

Allow unresolved as a separate outcome. A reviewer may be unable to establish the authoritative renewal date or determine whether a relevant plan existed. Those records should remain visible in the study. Removing difficult cases can make the system appear more reliable precisely because its most important limitations disappeared from the denominator. Report how many cases were unresolved and why.

Count four outcomes and keep their meanings separate

A true positive is a flagged record that the reviewer confirms needed action. A false positive is a flagged record that did not need the proposed action. A false negative is an unflagged record that did need action. A true negative is an unflagged record that did not. These labels depend on the operational definition; they are not judgments about whether the customer was happy or whether the contract renewed.

Precision is confirmed actionable alerts divided by all reviewed alerts with a resolved label. Recall is detected actionable cases divided by all actionable cases found in the reviewed sample. Report the raw counts with the ratios. If the sample deliberately overrepresents certain cohorts, explain that design and avoid presenting the unadjusted ratio as a portfolio-wide estimate. Small samples are useful for debugging but weak foundations for broad marketing claims.

Measure what happened after the alert

Detection is only the first stage. Track whether the assigned owner accepted the work, whether the underlying evidence was corrected, whether customer contact occurred and whether a specific next step was recorded. Preserve the timestamps so the team can distinguish prompt action from a task that remained open until the renewal passed. Choose a review window that fits the operating rhythm rather than the desired result.

Task completion should be inspected rather than assumed to mean resolution. Some tasks are closed because they are duplicates, incorrectly routed or irrelevant. Others are closed after useful outreach but before the customer responds. Record these reasons separately. A high completion rate can conceal either a productive queue or a team that clears administrative clutter as quickly as possible.

Establish the counterfactual before claiming impact

To claim incremental retention, the team needs a credible comparison with what would have happened without the intervention. A before-and-after chart alone can reflect customer mix, seasonality, pricing changes, product improvements, staffing or the timing of renewals. Those factors may explain a change even when the alerting system performed exactly as designed.

A practical early report can stop short of causal claims. State how many confirmed gaps were detected, how many received an owned action and how many remained unresolved. Report contract values as exposure associated with the reviewed cohort, with currency and period definitions. Do not relabel exposure as revenue saved. If a later evaluation uses random assignment or a comparison cohort, document the design and its limitations before interpreting the difference.

Learn from bounded vendor evidence

Smarsh's September 3 support-AI release is useful because it names an observed interaction sample and describes its manual confidence assessment. Those disclosures help readers understand what was measured. They do not turn a support-deflection result into evidence about renewal retention, and they do not establish that another company's implementation will reproduce it. The lesson is to disclose the measurement boundary, not to borrow the headline percentage.

Use the same discipline in an internal renewal study. Name who supplied the data, who reviewed it, how the labels were defined and which outcomes were omitted. Keep evaluator judgment separate from observed system events. If the company selling the tool also conducts the evaluation, disclose that relationship. Independence is a property of the study design, not a tone of voice in the report.

Publish a result that can be challenged

The final report should include population, unit, dates, sample selection, exclusions, definitions, reviewer method, raw outcome counts, unresolved cases, action follow-through and known sources of bias. Link a redacted example showing how one label was reached. Retain enough detail for another reviewer to reproduce the classification without exposing customer information.

Treat disagreement as useful evidence. If reviewers differ on whether a plan existed, the operating definition may need work. If most false positives come from one association pattern, fix that pattern and evaluate again using a new version. If false negatives cluster around a particular renewal source, narrow the product claim until coverage improves. A precise limitation is more valuable than a broad success claim that collapses under inspection.