From Impression Logs to Trustworthy A/B Tests

From Impression Logs to Trustworthy A/B Tests

Exposure logging gives a recommender system the denominator it was missing.

It tells us what the system showed, in which order, in what context, and under which request. That is the foundation I described in Impression Logs and Exposure Data: the Missing Half of Recommender Evaluation.

But an exposure log does not automatically make an experiment trustworthy.

An A/B test can still lie when:

• assignment changes between requests
• the wrong users enter the experiment
• treatment and control are logged differently
• assignments fail to join to exposures
• outcomes join to the wrong request
• one variant changes whether logging runs
• traffic allocation does not match the configured ratio
• the analysis silently drops difficult rows

My rule is:

Treat eligibility, assignment, exposure, and outcome as separate events joined by explicit identities.

The experiment result is only as trustworthy as that chain.

The four events answer different questions

I keep four concepts separate.

Eligibility answers:

text
Could this subject have entered this experiment at this moment?

Assignment answers:

text
Which variant did the randomization policy choose?

Exposure answers:

text
What intervention did the product actually render or deliver?

Outcome answers:

text
What happened afterwards?

They are related, but they are not interchangeable.

A user can be assigned to treatment and never open the page. A recommendation request can be made but fail before rendering. A rendered list can receive no click. An outcome can arrive after the experiment stops.

Merging these states into one "experiment event" makes the happy path easy and every investigation difficult.

Deterministic assignment is the boring default

For many product experiments, I want the same subject to receive the same variant for the lifetime of the experiment.

A deterministic hash assignment does that without storing a mutable row for every decision.

Here is a simplified Go implementation with two independent buckets: one for traffic inclusion and one for variant assignment.

go
package experiment

import (
	"crypto/sha256"
	"encoding/binary"
)

const basisPoints = uint64(10_000)

func bucket(value string) uint64 {
	sum := sha256.Sum256([]byte(value))
	return binary.BigEndian.Uint64(sum[:8])
}

func Assign(
	experimentID string,
	subjectID string,
	salt string,
	trafficBPS uint64,
) (variant string, assigned bool) {
	if trafficBPS > basisPoints {
		trafficBPS = basisPoints
	}

	inclusionKey := "inclusion|" + salt + "|" + experimentID + "|" + subjectID
	if bucket(inclusionKey)%basisPoints >= trafficBPS {
		return "", false
	}

	variantKey := "variant|" + salt + "|" + experimentID + "|" + subjectID
	if bucket(variantKey)%2 == 0 {
		return "control", true
	}

	return "treatment", true
}

The exact hash function is not the interesting part. The operational properties are:

• assignment is stable for the same subject, experiment, and salt
• changing traffic allocation does not reshuffle existing included subjects between variants
• variant assignment is independent from inclusion
• the algorithm is versioned and testable
• the subject identifier is stable at the chosen randomization unit
• the salt is controlled configuration, not regenerated on restart

I include an assignment_version in the event. If the algorithm or salt changes, I want the data to admit it.

json
{
  "experiment_id": "home-ranker-v2",
  "subject_id_hash": "usr_7d9...",
  "variant": "treatment",
  "assignment_version": "hash-v1",
  "traffic_bps": 2500,
  "assigned_at": "2026-08-20T10:00:00Z",
  "request_id": "req_01J..."
}

Choose the randomization unit before writing code

A recommender experiment can randomize by:

• user
• account
• household
• device
• session
• request
• region or cluster
• time window

The unit changes what the estimate means.

Request-level randomization may expose one user to both variants and create carryover. User-level randomization may be wrong when several users share an account and affect one another. Marketplace, advertising, and social systems can have interference where one participant's treatment changes another participant's outcomes.

There is no universal unit. There is only a unit aligned with the mechanism and the decision.

For a normal personalized feed where the goal is a stable experience, a pseudonymous user or account identifier is often a sensible starting point. I record the choice in the experiment definition rather than leaving it implicit in helper code.

Assignment must be logged even when exposure fails

Suppose treatment triggers an error before the recommendations render.

If assignment is logged only inside the successful render path, treatment's failures disappear from the experiment population. Control appears healthier partly because the data collection excluded treatment's broken requests.

So the assignment event occurs at the decision boundary, before the intervention.

The exposure event occurs only when the product's defined exposure condition is met.

json
{
  "request_id": "req_01J...",
  "experiment_id": "home-ranker-v2",
  "experiment_variant": "treatment",
  "subject_id_hash": "usr_7d9...",
  "surface": "home",
  "rendered_at": "2026-08-20T10:00:00.120Z",
  "items": [
    {"item_id": "item_42", "rank": 1},
    {"item_id": "item_17", "rank": 2}
  ]
}

The assignment-to-exposure gap is not missing data to quietly delete. It is a diagnostic signal.

Analyse by assignment; diagnose with exposure

For the primary causal comparison, I usually begin with intent to treat: analyse subjects according to their randomized assignment.

That preserves the randomization.

Filtering the primary result to only subjects who were exposed can introduce bias when the treatment itself changes exposure probability. A treatment that crashes before rendering would conveniently remove its worst cases from an exposure-only analysis.

Exposure data remains essential. I use it to:

• verify the intervention was delivered
• measure assignment-to-exposure join health
• understand dose and surface behaviour
• debug rendering and client instrumentation
• compute recommendation metrics such as click-through per rendered slate
• investigate heterogeneous effects

The distinction is subtle but practical:

text
Assignment answers the causal experiment question.
Exposure answers whether and how the product delivered the treatment.

A mature analysis may estimate treatment-on-the-treated effects with stronger assumptions. I do not start there merely because the exposure table is convenient.

The join model should be explicit

In RecSys, the core contract is:

• exposure: request_id, pseudonymous user, timestamp, displayed items
• outcome: request_id, user, item, event type, timestamp
• assignment: experiment_id, variant, request_id, user, timestamp

The repository documents this in the RecSys data contracts.

request_id reconstructs one recommendation interaction.

experiment_id and subject identity reconstruct assignment over the experiment lifetime.

item_id connects item-level outcomes to a displayed slate.

I avoid joining only on user and a broad time window. That can attribute a click to the wrong recommendation request, especially when a user refreshes or sees several surfaces.

A join policy should define:

• attribution window
• whether one outcome can attach to multiple exposures
• which exposure wins when several qualify
• how late events are handled
• whether render time or serve time begins the window
• how duplicate client events are deduplicated
• which clock is authoritative

These are product semantics disguised as SQL.

Measure join health before treatment effect

Before I look at conversion lift, I compute data-quality measures by variant.

text
assignment_count
unique_assigned_subjects
assignment_to_exposure_join_rate
exposure_to_outcome_join_rate
duplicate_assignment_rate
duplicate_exposure_rate
missing_request_id_rate
unknown_variant_rate
late_outcome_rate

The important phrase is by variant.

A 98% join rate can look respectable while hiding 99.5% in control and 96.5% in treatment. That asymmetry can bias the result.

I also compare event rates by client version, surface, platform, country, and deployment. Instrumentation bugs often respect software boundaries more faithfully than experiment boundaries.

Sample ratio mismatch is a fire alarm

If a 50/50 experiment assigns 510,000 subjects to control and 490,000 to treatment, the question is not whether the difference "looks small."

The question is whether it is plausible under the randomization model.

A sample ratio mismatch, or SRM, occurs when observed assignment counts differ materially from the expected allocation. It can reveal:

• biased eligibility
• inconsistent hashing
• dropped events
• variant-specific failures
• bots or internal traffic entering one arm
• filtering after assignment
• broken ramp configuration
• subject identity instability

Microsoft's experimentation researchers describe SRM as a high-value trustworthiness check and provide a practical taxonomy in Diagnosing Sample Ratio Mismatch in Online Controlled Experiments.

For two variants, the basic chi-squared statistic is straightforward:

text
expected_control = total * configured_control_share
expected_treatment = total * configured_treatment_share

chi_squared =
    (observed_control - expected_control)^2 / expected_control
  + (observed_treatment - expected_treatment)^2 / expected_treatment

The platform should compute the p-value and alert at a deliberately chosen threshold.

I do not "correct" an unexplained SRM by reweighting the final metric and moving on. The mismatch is evidence that the data-generating process may not be what I think it is.

A basic assignment-level conversion query

For a simple binary outcome, the analysis population begins with assignments.

In this example, an attribution step has already produced experiment_outcomes by joining raw outcomes to the relevant assignment/request and applying the documented attribution window. That derived table should remain reproducible from the raw events.

sql
WITH assigned AS (
    SELECT DISTINCT
        experiment_id,
        subject_id_hash,
        variant
    FROM experiment_assignments
    WHERE experiment_id = 'home-ranker-v2'
),
converted AS (
    SELECT DISTINCT
        experiment_id,
        subject_id_hash
    FROM experiment_outcomes
    WHERE experiment_id = 'home-ranker-v2'
      AND event_type = 'subscription_started'
)
SELECT
    assigned.variant,
    count(*) AS assigned_subjects,
    avg(
        CASE
            WHEN converted.subject_id_hash IS NOT NULL THEN 1.0
            ELSE 0.0
        END
    ) AS conversion_rate
FROM assigned
LEFT JOIN converted
    ON converted.experiment_id = assigned.experiment_id
   AND converted.subject_id_hash = assigned.subject_id_hash
GROUP BY assigned.variant;

This is not a complete statistical analysis. It does not provide uncertainty intervals, variance reduction, multiple-testing control, or protection against repeated peeking.

It demonstrates the population: all assigned subjects, including those with no outcome.

For recommendation-specific metrics such as click-through rate, I also calculate exposure-level measures with the slate and position available. Those answer a different question and should be labelled accordingly.

Guardrails belong beside the primary metric

An experiment can improve clicks by making the product slower, noisier, or less diverse.

I define guardrails before launch:

• request errors
• latency
• empty recommendation rate
• timeouts
• client rendering failures
• unsubscribe or complaint rate
• content diversity or concentration
• downstream conversion
• resource cost
• warning and policy-violation counts

The exact set follows the plausible harm.

In the RecSys experiment runbook, launch checks include assignment fields, exposure joins, errors, latency, warnings, and empty recommendations. The runbook says to hold when assignment joins, sample ratio, or schemas are not trustworthy. That is the right order: do not debate lift while the measuring instrument is broken.

Ramp traffic to validate the system, not to manufacture significance

A cautious ramp might be:

text
internal or synthetic traffic
  -> 1%
  -> validate assignment and event flow
  -> 10%
  -> validate guardrails and sample ratio
  -> 25%
  -> continue until planned allocation

At each stage I check:

• configured and observed traffic
• stable assignment across repeat requests
• SRM
• join rates by variant
• error and latency guardrails
• exposure counts
• unexpected empty variants
• rollback or disable lever

The ramp is not an excuse to inspect the primary metric every hour and stop when it becomes flattering. Sequential analysis needs an explicit method. Operational checks and causal decision rules are different things.

Keep an evidence bundle

For every experiment, I retain:

text
experiment definition
assignment algorithm and version
salt/configuration identifier
eligibility rules
start and stop time
traffic changes
deployment versions
assignment counts
SRM result
join-health report
metric definitions
guardrail results
analysis code or query hash
decision
rollback or rollout action

This makes the result reproducible after the dashboard has moved on.

It also makes failed experiments valuable. A held experiment with a clearly diagnosed instrumentation fault teaches the platform something. An unexplained dashboard screenshot teaches mostly humility.

The result should survive one hostile question

Before trusting an A/B result, I ask:

What plausible logging or assignment bug could create this apparent lift without the treatment actually helping?

Then I try to disprove that explanation with evidence.

That question is not pessimism. It is the experiment equivalent of testing a rollback before needing one.

Exposure logging gives recommender evaluation an honest denominator. Deterministic assignment gives the intervention a stable identity. Join checks and SRM tell us whether the evidence path remained intact.

Randomization earns trust only when the system instruments it from eligibility to outcome.