Chapter 05

Querying exceptions

Now that Twig's engineers capture high quality exceptions, they can query them for analysis.

Here are ways Twig's engineers use other PostHog events to understand the impact of issues in error tracking.

Each check below has a prompt to give an agent on Learn Error Tracking with your agent.

Join exceptions to the rest of Twig's data

PostHog keeps Twig's events in one context warehouse with persons, groups, sessions, and the tables an app can sync from other tools, like Zendesk. HogQL queries all of it together, so anything sent to PostHog can be joined to Twig's exceptions.

The engineers want to know which issue to fix first. The exception alone can't tell them, but each one carries the same context as Twig's other events:

  • Which guest hit it, in distinct_id and their person properties.
  • Which host's stay it was, in $groups.
  • Which session it happened in, in $session_id, which links to the replay.
  • Which page it happened on, in $current_url.
  • Which flag variant the guest saw, in $feature/<flag-key>.

Using these clues, Twig's engineers can piece together a more complete story.

Issues and exceptions lists every property PostHog sets.

Rank issues by bookings at risk

Hosts pay Twig a monthly fee, which the engineers sync from Stripe as each host's mrr. They rank issues by the MRR of the hosts they hit. An issue that hits high-MRR hosts can matter more, even with fewer occurrences Fig. 1.

Book stay fails · 620 exceptions

$48,000

Payout statement fails · 180 exceptions

$16,000

Stay search times out · 9,000 exceptions

$4,000

mrr

Fig. 1 – Example: issues ranked by the MRR of the hosts they hit. Stay search throws the most exceptions, but Book stay fails for hosts who pay twelve times as much.

This joins exceptions to hosts on $groups:

SourceWhat it adds
$exceptionThe hosts each issue hit
groupsWhat each host pays, from its mrr property

The Errors in paying hosts scout runs a related check every day. It reports issues concentrated in paying hosts.

Check whether an error makes guests leave

To see whether an error makes guests leave, the engineers check three things:

  • Funnels. The engineers build a funnel of booking a stay on Twig, with its graph type set to historical trends, to see whether conversion fell when the error appeared Fig. 2. They break it down by the issue's cohort, created from the issue page, to compare people who hit the error with people who didn't.
  • Retention. They filter a retention insight to the issue's cohort and compare it with all users. If people who hit the error come back less, the error is driving churn.
  • Session drop-off. They check whether people did anything within a minute of each error. Some sessions end anyway, so they compare each issue with the overall drop-off.
Error first seen

300 fewer confirmed bookings per day

Clicked Book stay: 1,000/dayConfirmed: 800/day to 500/day

Fig. 2 – Example: Twig guests click Book stay as often as before, but the share that confirms a booking falls from 80% to 50% when the error appears.

The session drop-off check joins exceptions to the events that followed on $session_id:

SourceWhat it adds
$exceptionWhen the error fired
$pageview and Twig's named events, like twig_stay_viewedWhether the guest kept going

Watch what guests saw

When an error makes guests leave, the engineers watch their sessions to see what happened.

This joins exceptions to recordings on $session_id:

SourceWhat it adds
$exceptionThe error and when it fired
session_replay_eventsWhat the person saw and did

Start from one host's errors

Triage can also start from a host. When a host reports a problem, the engineers rank every issue on that host's stays by how many guests it impacts Fig. 3. In the issue list, that's the host as a filter, sorted by Users.

Book stay fails · 314 exceptions

186

Payout statement fails · 41 exceptions

27

Stay search times out · 2,840 exceptions

2

users

Fig. 3 – Example: one host's issues, ranked by how many of the host's guests each one hit. Stay search fired 2,840 times for only 2 guests, which looks like a retry loop.

This joins exceptions to hosts on $groups:

SourceWhat it adds
$exceptionEach issue and how many guests it impacts
groupsWhich host's stays those guests were on

Find the errors behind support tickets

Guests and hosts reach Twig's support in Zendesk, which the engineers sync to PostHog. A ticket with an error behind it points at a bug they can fix.

This joins tickets to exceptions on the session ID in each ticket's replay link:

SourceWhat it adds
A tickets table, like zendesk.ticketsWhat the customer reported
$exceptionThe error behind it

Reach the guests an error hit

Once the engineers fix an issue, they tell the guests it hit, or survey them. They create a cohort from the issue page, then duplicate it as a static cohort to target them with a survey or flag.

This joins exceptions to people on distinct_id:

SourceWhat it adds
$exceptionThe issue each person hit
personsWho they are, so the team can reach them

Catch new errors in experiments

When Twig ships a change as an experiment, the engineers add $exception as a secondary metric to compare errors across variants.

To see when errors started, they chart each variant's error rate Fig. 4. With an uneven split, they compare a ratio, like exceptions per pageview, so a variant on 5% of traffic compares fairly with one on 95%. The Feature Flags pocket guide covers running the experiment.

Booking form deploy

Control

2% of sessions hit an error

New booking form

2% to 8% of sessions hit an error

Fig. 4 – Example: after a deploy, Twig's new booking form's affected-session rate rises to 8% while control stays at 2%. Both lines use the same scale.

This joins exceptions to variants on $feature/<flag-key>:

SourceWhat it adds
$exceptionErrors in each variant
$pageviewTraffic in each variant, for the rate

Track crash-free sessions

Twig is a web app, so the engineers track stability as the share of sessions without an unhandled exception.

This joins sessions to unhandled exceptions on $session_id:

SourceWhat it adds
sessionsEvery session
$exception with $exception_handled set to falseWhich sessions hit an unhandled exception

To rank your issues by the revenue they put at risk, give an agent the revenue skill Fig. 5.

error-tracking-errors-by-revenue/SKILL.md
---
name: error-tracking-errors-by-revenue
description: >
Build a PostHog insight that joins exceptions to your accounts and revenue, such as the new
errors ranked by the MRR of the accounts they hit. Find the events, find the key that joins it,
pick a chart, then adapt the example query to this project.
---
# Rank errors by what they put at risk
Read the current docs first. Every PostHog docs URL returns Markdown if you append `.md`:
- https://posthog.com/docs/data-warehouse/sql.md
- https://posthog.com/docs/product-analytics/group-analytics.md
- https://posthog.com/docs/data-warehouse/warehouse-properties.md
If the user hasn't said what to rank by, ask: MRR, ARR, plan, or seats. Then work through these
steps in order.
## 1. Find the events and properties
You need three things. Check that each one exists in this project before you write a query.
- **Exceptions.** `$exception` events. Each one has an `issue_id`, and
`properties.$exception_list[1]` holds the error's type and message.
- **The account each exception came from.** Group types are defined by each project, so list this
project's group types and find the one that represents an account, such as `company`. Note its
index. If the project has no groups, look for an account property on persons, such as a company
ID or email domain.
- **What each account is worth.** A property on that group, such as `mrr`, `arr`, or `plan`. Check
that most accounts have a value.
If no account has a value, stop. Tell the user that warehouse properties can sync one from Stripe
or their database. Don't guess one.
## 2. Find the key that joins them
Each event's `$groups` property maps the project's group types to keys, such as
`{"company": "acme"}`. In HogQL, the same keys are the columns `$group_0` to `$group_4`, and the
index from step 1 picks the column.
- Check that most `$exception` events from the last 7 days have that key. If they don't, the SDK
doesn't set the group when it captures exceptions. Tell the user, and point them to group
analytics for their SDK.
- Join exceptions to the `groups` table on `groups.key` and `groups.index`.
- Count each account once per error. An account that hits an error 500 times is still one account.
## 3. Pick the shape
- One value per error reads best as a bar chart. In the SQL editor, open **Visualization**, pick
**Horizontal bar chart**, and put the error label on the X-axis and the value on the Y-axis.
- Label each bar with the exception type and the start of its message. Issue IDs are unreadable on
an axis.
- Show the top 10.
- Match the time window to the question. "New errors" means issues created in the last 7 days, from
`system.error_tracking_issues`. To rank every error, drop that filter.
- Group by the issue ID as well as the label. Two issues can share a label.
## 4. Adapt the example
This query ranks the errors first seen in the last 7 days by the MRR of the accounts they hit. It
assumes the account group type has index 0 and an `mrr` property. Change the group index, the
property, and the time windows to match what you found in steps 1 and 2. Don't run it unchanged.
```sql
WITH issues AS (
SELECT
id AS issue_id,
concat(name, ': ', substring(description, 1, 45)) AS error
FROM system.error_tracking_issues
WHERE created_at >= now() - INTERVAL 7 DAY
),
affected AS (
SELECT DISTINCT issue_id, $group_0 AS account
FROM events
WHERE event = '$exception'
AND timestamp >= now() - INTERVAL 7 DAY
AND notEmpty($group_0)
AND issue_id IN (SELECT issue_id FROM issues)
)
SELECT
i.error AS error,
round(sum(toFloat(g.properties.mrr))) AS mrr_at_risk
FROM affected a
JOIN issues i ON i.issue_id = a.issue_id
LEFT JOIN groups g ON g.key = a.account AND g.index = 0
GROUP BY i.issue_id, i.error
ORDER BY mrr_at_risk DESC
LIMIT 10
```
Run it and check the result. The values shouldn't all be zero, and the labels should be readable.
Then save it as an insight.
## Finally
Report the top three errors and what they put at risk. Say that the value is what those accounts
pay, not revenue the error lost.
Fig. 5 – The skill that ranks errors by the revenue they put at risk. Copy it and give it to your agent.