Testing and launching an experiment

Once you've written your code, it's a good idea to test that each variant behaves as you'd expect. If you find out your implementation had a bug after you've launched the experiment, you lose days of effort as the experiment results can no longer be trusted.

The best way to do this is adding an optional override to your release conditions. For example, you can create an override to assign a user to the test variant if their email is your own (or someone in your team). To do this:

1. Go to your experiment feature flag.

2. Ensure the feature flag is enabled by checking the "Enable feature flag" box.

3. Add a new condition set with the condition to email = your_email@domain.com. Set the rollout percentage for this set to 100%.

   • In cases where email is not available (such as when your users are logged out), you can use a parameter like utm_source and append ?utm_source=your_variant_name to your URL.

4. Set the optional override for the variant you'd like to assign these users to.

5. Click "Save".

Once you test it works, you're ready to launch your experiment.

Notes:

• The feature flag is activated only when you launch the experiment, or if you've manually checked the "Enable feature flag" box.
• While the PostHog toolbar enables you to toggle feature flags on and off, this only works for active feature flags and won't work for your experiment feature flag while it is still in draft mode.

Viewing experiment results

While the experiment is running, you can see results on the details page:

This page shows:

• Status, whether draft, running, or complete as well as whether it has reached significance or not.
• Data collection progress based on an estimated target number of participants and minimum detectable effect.
• Win probability for each variant.
• Statistical significance for the results.
• Variants, their release conditions and distribution.
• Primary and secondary metrics.

Click View recordings to see session recordings of experiment participants. By default, all metric events are applied as filters to the playlist meaning you'll only see recordings for users who completed the funnel or fired the trend metric. However, the filters don't map exactly to the statistical calculations (funnel attribution type isn't applied, for instance), so the recordings don't map exactly to the experiment results.

Ending an experiment

After you've analyzed your experiment metrics and you're ready to end your experiment, you can click the Ship a variant button on the experiment page to roll out a variant and stop the experiment. This button only appears when the experiment has reached statistical significance and its feature flag is not rolled out to 100% for any variant yet.

If you want more precise control over your release, you can also set the release condition for the feature flag and stop the experiment manually.

Beyond this, we recommend:

1. Sharing the results with your team.

2. Documenting conclusions and findings in the description field your experiment. This helps preserve historical context for future team members.

3. Removing the experiment and losing variant's code.

4. Archiving the experiment.

Further reading

Want to learn more about how to run successful experiments in PostHog? Try these tutorials:

• A software engineer's guide to A/B testing
• 8 annoying A/B testing mistakes every engineer should know
• When and how to run group-targeted A/B tests