Chapter 07

Find AI failures that return a 200

Your error rate is a flat green line. LGTM. Also, your AI summarizer has been quietly dropping the last section of every long document for two weeks.

Both of those are true at the same time, and that's the trouble with error rates – they count the failures your code noticed. They say nothing about the ones it didn't.

Your error rate only counts what threw

PostHog Error Tracking works if you capture exceptions. Set $ai_is_error and it normalizes the message, so Request req_abc123 failed and Request req_xyz789 failed collapse into one issue with counts for traces, users, and days seen. This catches your timeouts, your 429s, and the malformed response that blew up your parser.

Error Tracking alone can't catch any of this:

The model ignored an instruction
It answered from the wrong document
A tool came back empty and it carried on regardless
Its context window ran out and stopped three quarters of the way through

Every one of those can return 200 with a plausible-looking body.

Which is how you end up with the Fig. 1 scenario: an error rate that looks normal, and the failure nobody is paying attention to.

prompt template change

Error rate

flat at ~0.4%

Summaries missing the last section

4% → 19%

Fig. 1 – Two weeks of the summarize workflow. One of these lines is on your dashboard.

Which traces to read

You can't possibly read every trace yourself, so you need a filter. Here's some examples of filters, and how each would have done against your summarizer's failure:

Errors. Nothing – it's hidden.
A failed evaluation. Only if someone already wrote an eval asking "did this cover the whole document?" You can only catch what you thought to check.
Negative user feedback. You may get a few, from the people who noticed and bothered to say so.
Token outliers. Sort by output tokens letting you find generations that stop on the same number of tokens, which shares a common failure mode like hitting your output limit.
Trace clusters. Won't necessarily find it, but it will stop you reading five copies of the same trace.

So sort by output tokens and take the top of the pile. You now have a reading list, but you still don't know what's wrong.

Group by what broke first

The top of that pile is 22 summarize traces. You read them and fourteen come back with the last section of the document missing from the summary. The other eight are long documents that summarized fine.

Each of these broken traces will exhibit different behavior. One user said it cut off mid-sentence but another said it skipped the risks section entirely. An eval failed on "does this mention the deadline?"

These failure modes look different but are the same underlying issue. To see the commonality, you need the earliest thing that went wrong in each trace.

Anatomy of a trace

So open a trace – Fig. 2.

$ai_tracesummarize_document200 OK · 4.1s · $0.031
$ai_spanload_document11,400 tokens in
$ai_generationsummarize_section_1stop: end_turn · 780 out
$ai_generationsummarize_section_2stop: end_turn · 812 out
$ai_generationsummarize_section_3stop: max_tokens · 1,024 out

Fig. 2 – One of the 14. Every generation succeeded, and the summary still comes back short.

The document went in at 11,400 tokens. Sections one and two came back clean, end_turn on both. Section three stopped on max_tokens.

There it is. This wasn't the prompt, the model, or retrieval. The output budget ran out mid-section and the code shipped whatever it had.

But nothing threw, so the only evidence you have is one word in a stop reason, and you had to open the trace to find it.

Automate the traces

You aren't going to read 25 traces every morning – so hand it to a scout. It reads traces on a schedule, up to 25 in a run, and only writes when a failure mode shows up across independent requests.

When it does, you get a report with the workflow, the comparison window, what broke first, how often it happens against the whole population, and an action to take. That report will look something like Fig. 3.

Inbox

1 unread

Scout · error patterns6h ago

AI error pattern: the summarizer drops the last section on long documents

Across 22 traces read from the summarize workflow, 14 ended with the final document section missing from the output. Every one carried a max_tokens stop reason on the last generation and returned HTTP 200, so none of them appear in the error rate.

The pattern is confined to inputs over roughly 8k tokens and has grown from ~4% to ~19% of that segment since the prompt template changed on the 11th. Shorter documents are unaffected.

~19% of summarize requests over 8k tokens

Suggested action:

Raise the output token budget for long inputs, or chunk the document and summarize per section, and add an eval that fails when the last section is missing.

An agent opens the pull request. You review and merge.

Fig. 3 – What lands in your inbox when the scout finds it.Hover over the figure to learn about each element.

From there, you can fix it yourself, or choose to have PostHog open a PR to fix it for you.

Add the scout

The scout itself is just a file an agent follows on a schedule, and Fig. 4 is the one this guide is about.

signals-scout-ai-observability-error-patterns/SKILL.md
---
name: signals-scout-ai-observability-error-patterns
description: >
  Finds new or growing AI failure modes, including silent quality failures, and validates each
  pattern against real traces.
scout-tags:
  - ai-observability
---

# AI observability error patterns

Find new, growing, or recurring AI failure modes in the most recent complete 24 hours. Compare them with the preceding 24 hours and the recent 7-day baseline.

The main signal is a recurring failure mode within one use case, validated by real traces and tied to a concrete next action. Raw error counts are pointers to investigate, not findings. Important AI failures can return HTTP 200 with no exception.

## Use the packaged analysis skills

Load these preinstalled skills through the runtime's packaged-skill mechanism when relevant:

- `exploring-ai-failures`
- `exploring-llm-traces`
- `exploring-llm-clusters`
- `exploring-llm-evaluations`
- `querying-posthog-data`

These are packaged runtime skills, not project skill-store entries. Do not use `skill-list` or `skill-get` to load them.

## Avoid duplicate work

Read this Scout's last 14 days of run summaries with `scout-runs-list`, filtered by its exact `skill_name` and current `skill_version`. Retrieve relevant details with `scout-runs-retrieve`.

Search the scratchpad and recent Inbox reports for the use case, failure mode, error, evaluation, cluster, and suspected cause. If a live report already covers the same pattern, add only materially new evidence with `scout-edit-report`. Skip it when the evidence and impact are unchanged.

## Find and read failures

Work on one use case at a time because different trace types fail differently.

1. Discover the project's trace taxonomy from properties that actually exist, such as feature, workflow, model, or span name.
2. Choose the use case with the clearest recent change or the stalest prior coverage.
3. Select traces using the signals that fit the data: explicit errors, tool failures, retries, evaluation failures, negative feedback, latency or token outliers, semantic clusters, or a stratified sample.
4. Open and read real traces. Queries choose what to read but do not establish a failure mode.
5. Identify the first thing that went wrong in each failing trace and group traces by that root failure instead of downstream symptoms.
6. Continue until the common modes stabilize, with a maximum of 25 traces in one run.
7. Quantify each mode against the full use-case population. Keep sample counts and population estimates separate.

Look for loud errors and silent failures such as ignored instructions, wrong answers, missing context, tool misuse, loops, malformed output, or incomplete work. Do not infer a silent failure from text matching alone.

Minimize personal data. Summarize the relevant behavior and cite trace IDs or links without copying sensitive prompt or response content.

Close without a report when no repeated, materially changed, or actionable failure mode clears the bar.

## Report only actionable patterns

A report-worthy failure must appear across independent traces or have clear systemic impact, be new or materially worse than baseline, be validated by representative traces, and lead to a concrete action.

Create one report per root failure, not per trace or error string. Create no more than two reports per run. Search the Inbox again before writing. Edit a matching live report instead of creating a duplicate.

Title a new report `AI error pattern: <specific mode>`. Include the affected use case and comparison window, how often it appeared, what failed first, why it matters, and the best next action. Include one to three representative trace links or IDs.

Do not report expected cancellations, test or synthetic evaluation traffic presented as production impact, known provider incidents covered elsewhere, one-off traces without systemic impact, or error aggregates that were not validated by reading traces.

Finish with a short run summary covering what you reviewed, what you reported or updated, and what you ruled out.

Fig. 4 – The scout itself. The file an agent follows on a schedule.

The custom scout for this already lives in PostHog. You just have to enable it, and it's set to run daily at 9:00 AM.

You'll need these to run this scout.

RequiredAI Observability instrumented, so $ai_generation events are arriving
RequiredTraces carrying a feature, workflow, or span name, so failures can be grouped by use case
RecommendedOnline evaluations or user feedback, to surface failures that never error

Add this scoutOpens PostHog with it prefilled. Review it, then hit Create.

Not set up yet? One command installs PostHog, connects GitHub, and turns on a default troop of scouts. That troop doesn't include this one – add it above once you're set up.

npx @posthog/wizard self-driving