Validate assignment, exposure and outcome before reading experiment lift

This protocol does not calculate a universal benchmark for experimentation quality. It creates a testable chain from intended assignment to actual exposure and measured outcome. The output is a data-quality report with explicit exclusions, not a product-performance ranking. Use synthetic or approved test subjects first and keep customer identifiers out of the working file unless the review requires them.

Research question

Can the team reproduce, for a representative sample, which subject was assigned, whether the subject was exposed, which outcome events qualified and which account or commercial object would receive a downstream action? The protocol should answer that question across normal, duplicate, late, changed-identity and missing-data cases.

1. Freeze the experiment definition

Export or record the experiment ID, version, environment, variants, allocation, targeting rule, start, scheduled stop, actual stop, assignment unit, exposure rule, primary metric, guardrails, observation window and owner. Amplitude's September release describes scheduled stopping and a frozen analysis window. Capture those values before inspecting results so the research definition cannot move in response to the outcome.

If the platform supports several statistical methods, record the selected method and settings. Mixpanel's announcement lists frequentist, sequential and Bayesian options, with Bayesian described as beta, plus several health checks. Do not combine outputs from different methods in one number. The goal here is reproducibility, not choosing a method by whichever headline looks stronger.

2. Build the assignment population

Extract one row per assignment subject with subject key, experiment version, variant, assigned-at time, eligibility result and targeting attributes required to explain the decision. Deduplicate by the documented assignment key. If the product can evaluate both anonymously and after login, keep those identities separate until the merge rule is proven.

Select a stratified sample that includes every variant, relevant platform or region, several accounts with multiple users, newly created subjects, merged identities and records near the start or stop boundary. Record how the sample was chosen. A convenient sample of successful sessions will hide the cases that are most likely to break the join.

3. Prove qualifying exposure

For each sampled assignment, locate the event or server record that proves the variant was evaluated in the intended surface. Preserve source event ID, event time, ingestion time, application version, flag or experiment key and evaluation context. Classify rows as exposed, assigned-not-exposed, exposure-without-assignment, conflicting variant or unknown.

Repeated evaluations should not automatically create repeated subjects. Define whether first exposure, first exposure per session or another bounded rule starts the observation window. Test a network retry, offline delivery and a user who leaves before the surface loads. The platform dashboard may aggregate these cases; the research file should retain them.

4. Validate the outcome contract

Write the primary outcome as an event contract: event name and version, subject, qualifying properties, allowed source, observation window, duplicate rule and late-arrival treatment. Segment Protocols is one example of a tracking-plan approach. Whatever tool is used, the review should distinguish an event that arrived from one that represents the intended business action.

Sample raw outcome events and compare them with the product state they claim to represent. A completed onboarding event should match the required milestone state. A purchase event should reconcile to the authoritative order. A renewal or expansion outcome should not be inferred from a page view when the commercial record remains open or cancelled.

5. Join to account and revenue objects

Create an explicit bridge from the assignment subject to person, account or workspace, then to opportunity, subscription, contract or invoice only where the analysis needs that level. Preserve valid-from and valid-to dates for mutable relationships. Count rows before and after each join and list unmatched and multiplied subjects.

Do not force an account attribution when several accounts share a user or when a person changes company during the test. Unknown is a valid result. If one account contains subjects in different variants, define whether the account is contaminated, analyzed at user level or excluded from an account-level decision.

6. Reconcile the clock

Compare event time, ingestion time, assignment time, exposure time and commercial effective time. Re-run the result after the bounded late-event period and record every changed subject. If a source correction arrives after the result was approved, create a revised report with a new modified timestamp and decision note.

Use the same time zone and boundary semantics across platforms. A scheduled stop at midnight in one workspace and a warehouse date partition in UTC can create a silent extra or missing day. Test records exactly before, at and after the boundary.

7. Report quality before lift

Publish counts for eligible assignments, qualifying exposures, assigned-not-exposed subjects, exposure-without-assignment, variant conflicts, duplicate events, late events, unmatched identities, multiplied account joins and excluded records by reason. Show the denominator for every rate. These are internal diagnostic measures, not universal acceptable thresholds.

Only after this table is reviewed should the team interpret the experiment effect. If a quality issue is concentrated in one variant, platform or segment, a pooled result can be misleading. Route the defect to the tracking, identity or experiment owner and decide whether to correct, rerun, narrow or stop.

8. Reproduce one decision

Choose one sampled downstream action and trace it backwards: action record, commercial object, outcome event, qualifying exposure, assignment and experiment version. Then repeat the trace after changing one relevant state, such as removing eligibility, merging a test identity or stopping the flag. Confirm that the workflow fails safely and that the audit trail explains why.

The final deliverable is not a claim that the experiment worked. It is evidence that the measured population and the resulting operational action can be explained. That distinction keeps product learning useful without turning every dashboard movement into a revenue assertion.

Related reading: experiment evidence analysis · customer-event join validation · CRM data quality