Check whether the fix held

The Inbox validation scout checks each fix once it has reached guests, and reports the fixes that didn't hold. It waits and compares rates first, because some browsers keep running the old code for a while.

What the scout looks at

Three days ago at 10:30, the engineers merged a fix for the Trip details TypeError. The issue still fires at about the rate it did before the merge, so the scout reports it Fig. 1. When the rate fell hard and keeps falling, the fix is still rolling out, and the scout checks again later.

The report in your inbox

A fix that didn't hold becomes a new report in the team's inbox, linked to the original so the team can see what was already tried Fig. 2.

The scout itself

PostHog maintains this scout, listed as Inbox validation in the project's scouts. Its skill file is on GitHub. It waits 24 hours after the merge for a server fix, and 72 hours or more for web and mobile. Then it compares the issue's exceptions and affected users with the same window before the merge. It also checks Twig's overall traffic, so a quiet weekend doesn't pass for a fix. For a fix no report links to, the PR follow-up scout runs the same check. It runs every 24 hours by default and stays quiet unless something needs the team's attention.

Add this to your scout troop

PostHog ships this scout turned on, and the engineers can turn it off for each project. It checks a fix once exceptions reach PostHog and the fix's report resolves, either when its linked pull request merges or when the engineers resolve it themselves.