> AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt # Find AI failures that return a 200 – Self-driving pocket guide Canonical URL: https://posthog.com/docs/self-driving/pocket-guide/ai-error-patterns ## On this page - [Your error rate only counts what threw](#your-error-rate-only-counts-what-threw) - [Which traces to read](#which-traces-to-read) - [Group by what broke first](#group-by-what-broke-first) - [Anatomy of a trace](#anatomy-of-a-trace) - [Automate the traces](#automate-the-traces) Chapter 07 # Find AI failures that return a 200 – Self-driving pocket guide Your error rate is a flat green line. LGTM. Also, your AI summarizer has been quietly dropping the last section of every long document for two weeks. Both of those are true at the same time, and that's the trouble with error rates – they count the failures your code noticed. They say nothing about the ones it didn't. ## Your error rate only counts what threw PostHog Error Tracking works if you capture exceptions. Set `$ai_is_error` and it normalizes the message, so `Request req_abc123 failed` and `Request req_xyz789 failed` collapse into one issue with counts for traces, users, and days seen. This catches your timeouts, your 429s, and the malformed response that blew up your parser. Error Tracking alone can't catch any of this: - The model ignored an instruction - It answered from the wrong document - A tool came back empty and it carried on regardless - Its context window ran out and stopped three quarters of the way through Every one of those can return 200 with a plausible-looking body. Which is how you end up with the Fig. 1 scenario: an error rate that looks normal, and the failure nobody is paying attention to. prompt template change Error rate flat at ~0.4% Summaries missing the last section 4% → 19% Fig. 1 – Two weeks of the summarize workflow. One of these lines is on your dashboard. ## Which traces to read You can't possibly read every trace yourself, so you need a filter. Here's some examples of filters, and how each would have done against your summarizer's failure: - **Errors.** Nothing – it's hidden. - **A failed [evaluation](https://posthog.com/docs/ai-evals.md).** Only if someone already wrote an eval asking "did this cover the whole document?" You can only catch what you thought to check. - **Negative [user feedback](https://posthog.com/docs/ai-observability/collect-user-feedback.md).** You may get a few, from the people who noticed and bothered to say so. - **Token outliers.** Sort by output tokens letting you find generations that stop on the same number of tokens, which shares a common failure mode like hitting your output limit. - **Trace clusters.** Won't necessarily find it, but it will stop you reading five copies of the same trace. So sort by output tokens and take the top of the pile. You now have a reading list, but you still don't know what's wrong. ## Group by what broke first The top of that pile is 22 `summarize` traces. You read them and fourteen come back with the last section of the document missing from the summary. The other eight are long documents that summarized fine. Each of these broken traces will exhibit different behavior. One user said it cut off mid-sentence but another said it skipped the risks section entirely. An eval failed on "does this mention the deadline?" These failure modes look different but are the same underlying issue. To see the commonality, you need the earliest thing that went wrong in each [trace](https://posthog.com/docs/ai-observability/traces.md). ## Anatomy of a trace So open a trace – Fig. 2. $ai\_tracesummarize\_document200 OK · 4.1s · $0.0311$ai\_spanload\_document11,400 tokens in$ai\_generationsummarize\_section\_1stop: end\_turn · 780 out$ai\_generationsummarize\_section\_2stop: end\_turn · 812 out$ai\_generationsummarize\_section\_3stop: max\_tokens · 1,024 out2 Fig. 2 – One of the 14. Every generation succeeded, and the summary still comes back short. The document went in at 11,400 tokens. Sections one and two came back clean, `end_turn` on both. Section three stopped on `max_tokens`. There it is. This wasn't the prompt, the model, or retrieval. The output budget ran out mid-section and the code shipped whatever it had. But nothing threw, so the only evidence you have is one word in a stop reason, and you had to open the trace to find it. ## Automate the traces You aren't going to read 25 traces every morning – so hand it to a [scout](https://posthog.com/docs/self-driving/scouts.md). It reads traces on a schedule, up to 25 in a run, and only writes when a failure mode shows up across independent requests. When it does, you get a [report](https://posthog.com/docs/self-driving/reports.md) with the workflow, the comparison window, what broke first, how often it happens against the whole population, and an action to take. That report will look something like Fig. 3. Inbox 1 unread 1 Scout · error patterns·6h ago ### AI error pattern: the summarizer drops the last section on long documents Across 22 traces read from the `summarize` workflow, 14 ended with the final document section missing from the output. Every one carried a `max_tokens` stop reason on the last generation and returned HTTP 200, so none of them appear in the error rate. The pattern is confined to inputs over roughly 8k tokens and has grown from ~4% to ~19% of that segment since the prompt template changed on the 11th. Shorter documents are unaffected. ~19% of summarize requests over 8k tokens **Suggested action:** Raise the output token budget for long inputs, or chunk the document and summarize per section, and add an eval that fails when the last section is missing. An agent opens the pull request. You review and merge.