Chapter 06

Triage and alerts

Every week at stand-up, Twig's engineers go through the new issues together. At first it takes 5 minutes. As Twig grows, it takes 10, then 20, and eventually the whole meeting. Most of that time goes to the same questions every week: who owns this issue, and does it need attention now?

So the engineers look for ways to send each issue to the people who own it, and to only interrupt someone when an issue needs them.

Route issues to their owners with assignment rules

Most issues belong to one team. The Trips team owns everything under src/trips/, including Trip details. Instead of assigning those issues by hand at stand-up, the engineers add an assignment rule that sends issues from src/trips/ to the Trips team Fig. 1.

Rules run at ingestion, after source maps are applied, so they match Twig's unminified stack trace. The first rule that matches assigns the issue, so specific rules go first.

Track fixes in GitHub

Twig's engineering teams plan their sprints in GitHub issues. They configure Error Tracking to create mirrored issues in GitHub issues directly, so they can groom it as a part of their sprint planning Fig. 2.

If your team works out of GitLab, Linear, or Jira, you can do the same.

Get alerted when an issue spikes

Some issues can't wait for the next stand-up. When Trip details breaks for hundreds of guests, the Trips team wants to know right away, without a message for every new issue.

PostHog alerts on three triggers:

• Issue created fires for every new issue, which can mean many messages a day.
• Issue reopened fires when a resolved issue comes back, which catches regressions.
• Issue spiking fires when an issue gets far more exceptions than usual. PostHog compares each 5-minute window with the issue's baseline from the past hour Fig. 3.

The engineers only alert on spiking issues. New issues already go to the team that owns them, which grooms them, so an alert for each one only adds noise. Spike alerts go to the Trips team's Slack channel.

When the spike alert fires, someone on the Trips team opens the issue and watches a recording of a guest who hit it. If it's a real bug, they fix it and link the GitHub issue. If it's noise, they suppress it.

Stand-up takes 5 minutes again.

To set up the same for your app, create a role for each team in your organization settings, then add assignment rules and alerts in Error Tracking settings. Alerts can go to Slack or a webhook. PostHog AI can suggest rules and alerts from your recent exceptions:

Group my $exception events from the last 30 days by the source files in $exception_sources and the path of $current_url, and leave out third-party code. Name each group by the part of my product it covers, like trips or bookings. Suggest one error tracking assignment rule per group, with the property, operator, and value it filters on. List the rules most specific first, because the first rule that matches assigns the issue. Then suggest which error tracking alerts each team needs, for reopened issues and spikes, and where to send them.

Self-driving

As Twig grows with more users, checking alerts and dashboards and getting issues auto-assigned still leaves many exceptions unresolved. So Twig's engineers turn to self-driving to help resolve these issues.