Chapter 06
From flag to experiment
A rollout asks whether a change is safe. An experiment asks whether it's better. A flag with control and test variants splits people between two versions, but the split alone measures nothing. An experiment adds the parts that turn a split into an answer.
The four parts of an experiment
| Part | The question it answers | For new-checkout |
|---|---|---|
| Hypothesis | What do you expect to change, and why? | A one-page checkout will convert better than the three-step one |
| Eligible population | Who can be in it? | Signed-in people who reach checkout |
| Primary metric | What number decides it? | Checkout conversion rate |
| Exposure moment | When does a person count as in it? | When the checkout page decides which version to render |
Write the hypothesis and pick the primary metric before you start, so the result can't be read to fit whatever happened. The other two parts live on the flag.
The population lives in the release conditions
The experiment's flag decides who's eligible, the same way a rollout does. An experiment needs a multivariate flag with at least two variants, one of them named control Fig. 1. Use an existing flag that has those, with control as its first variant, or let the experiment create one.
A multivariate flag answers with a variant key instead of true or false.
Keep each person in one variant for the whole experiment. By default, a person exposed to more than one variant is left out of the results entirely, so every switch shrinks your sample. Switching happens when the distinct ID changes between checks. Call identify() before the first check, or turn on flag persistence across authentication so the variant survives login.
Exposure happens at the check
A person counts as exposed when a $feature_flag_called event comes in, and your flag checks already send it. Only exposed people count in the results, and anything they did before exposure is ignored.
So put the check where the experience splits, after every other condition. An earlier check counts people who never see the change as exposed, and they dilute your results. A check on the checkout page counts people who saw a checkout. A check on the home page counts everyone who visited.
Ship the winner, then remove the flag
When the primary metric has an answer, roll the winning variant out to everyone, and then clean up the flag: an experiment flag is temporary by design.