Experiments

An experiment splits the people who use a product between two or more versions of it, and measures which version does better against a metric chosen in advance. Experiments is also the name of the PostHog product that runs them.^[2]

For the statistical method and the reasons it is used, see A/B testing. This article describes the PostHog product.

What problem it solves[edit]

A team that ships a change and sees a number rise cannot tell whether the change caused the rise. The week, the season, a campaign, or an unrelated release could have caused it. Comparing before and after measures time as much as it measures the change.

An experiment removes that ambiguity by running both versions at once, on comparable groups of people. The difference between the groups is then attributable to the difference between the versions.

How it works in PostHog[edit]

An experiment is a feature flag with a metric attached.^[1]

1. Assignment. A multi-variant flag assigns each person to a variant. Assignment is stable per person, so nobody moves between variants mid-experiment.
2. Exposure. A person enters the analysis when the flag is evaluated for them. An experiment can instead use a custom exposure event, to count only the people who reached the part of the product being changed.
3. Metrics. A primary metric decides the result. Secondary metrics watch for damage elsewhere. A metric is a count, a sum, a ratio, or a retention measure, built on the same product analytics data everything else uses.
4. Result. PostHog reports the difference between variants with an interval around it, so a team can see how much of the result is noise.

How it relates to other PostHog products[edit]

See also

References

1. ^PostHog documentation, Experiments
2. ^PostHog Experiments

External links

• PostHog Experiments
• Experiments documentation