How to measure the evidence quality of return exceptions

Zendesk's September roundup highlights a Refundid integration that brings return information into the support ticket. This creates an opportunity to measure whether an agent can reconstruct a return case at the moment of customer contact. It does not create a public benchmark for response speed, refund accuracy or customer satisfaction. The purpose of this research design is to define a defensible local baseline: what proportion of sampled return conversations have the identity, status, decision and outcome evidence needed for a truthful reply and a later audit?

Define the unit before counting. Use one customer contact tied to one return case as the primary observation. Keep a stable ticket ID and return case ID, and record whether the ticket concerns several orders or several returned items. A single ticket can have multiple case units; a single case can have multiple contacts. Report both counts. Do not collapse them into one denominator, because a team that closes repeat contacts under a single ticket could appear to improve without solving the underlying problem.

Set an observation window, for example tickets opened during a named four-week period, and freeze it before analysis. Include all eligible return-related tickets in the source queue, not just those where the new sidebar rendered correctly. Record the exclusion rule for spam, test cases, non-merchant returns and records that cannot legally be inspected. Count missing order links and failed app loads as observable exceptions rather than dropping them. Preserve the extraction timestamp because return and payment states may change after the initial conversation.

For each sampled case, collect six fields: customer and order match evidence; item and quantity; return state and its source time; policy or exception decision and approver; refund or replacement instruction ID; and terminal state in the order or payment system. Add the support reply time and text, but do not use the text as the sole state source. The marketplace listing can tell an implementer what the integration claims to expose; the merchant must map those fields to its own order and finance records and document null or conflicting values.

The primary measure should be evidence completeness at first substantive reply. The numerator is the number of eligible case contacts for which a reviewer can locate a supported customer-order match, current return stage, source timestamp and a reply that does not outrun the source state. The denominator is all eligible case contacts in the window. Publish each component separately as well as the combined proportion. A single combined score hides whether the problem is identity, refresh delay, policy ambiguity or communication.

A second measure is authoritative outcome reconciliation. Among cases where a refund, replacement or exception was promised, count those with a terminal record in the system that actually owns the outcome and a stable link back to the ticket and case. Report unresolved cases separately instead of calling them failures before their normal processing window has elapsed. For completed cases, sample the promised amount, currency, item and destination against the final record. A matched word in a support note is not a settled financial event.

A third measure is correction burden. Count contacts reopened because of an incorrect order match, stale status, premature promise, policy reversal or duplicate action. Use mutually exclusive primary reason codes and allow secondary factors. Record when the mistake was detected and who corrected it. A lower reopening count could reflect poor tagging, so audit a random set of supposedly clean closures. The analysis should describe coding rules, reviewer training and disagreement resolution; otherwise the rate is too easy to move by changing labels.

Sampling should respect variation. Stratify by partial versus full return, one versus multiple items, standard versus exception policy, first versus repeat contact, and app-rendered versus app-unavailable cases. If volume permits, include different support queues and payment methods. Publish the sample size and actual selection method for each stratum, then weight back to the eligible population only when the necessary counts are known. Do not turn a small hand-picked audit into a site-wide percentage.

Use two reviewers for an initial calibration sample. Have each independently mark identity evidence, state timing, policy authority and final outcome, then compare disagreements. Resolve the coding guide before scoring the full sample. Retain the guide version and example decisions. Reviewers should not infer that a sidebar value is authoritative merely because it is visually prominent; they should identify its source and compare it with the designated system of record. This is particularly important for returns whose status changed after the ticket was answered.

If the team wants to estimate change after deployment, establish the pre-implementation baseline using the same case definition and source systems. Match on queue, return type, period and policy changes where possible. A simple before-and-after gap cannot establish causality if staffing, return volume, payment processing or policy also changed. Report those changes and avoid claiming the app caused an improvement without a credible comparison. The first useful result may be a map of missing evidence rather than a performance uplift.

Publish a compact methods box with the sponsor, systems queried, time window, eligible population, sample selection, exclusions, fields, stage definitions, reviewer process, missingness, limitations and extraction date. Report counts alongside percentages. If a group is small or contains sensitive customer details, aggregate or suppress it according to the organization's policy. State plainly that these are local process measures; Zendesk and Refundid have not supplied the observed values or a cross-company benchmark through the cited announcement.

The decision from the study should be operational. If matches are weak, repair identity linking before encouraging faster replies. If statuses are stale, expose source time and an alternate lookup. If policy exceptions are undocumented, create an owner and decision record. If terminal outcomes cannot be reconciled, join ticket, case, order and payment identifiers. Rerun the same method after the specific repair. A measurement system earns trust when it points to a correctable failure mode, not when it produces an attractive percentage. Related reading: Benchmarks · Customer success.