> AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt # Debug a wrong answer with traces – AI Observability pocket guide [](/pocket-guides.md)Aa [](/pocket-guides/ai-observability/throw-away-answers.md) # Debug a wrong answer with traces – AI Observability pocket guide Your docs assistant tells a user to authenticate with a header your API dropped two versions ago. Everything looks normal and the call succeeded. So why did it say that? "The model hallucinated" is a guess that might send you off to rewrite a prompt that doesn't need changing. This use case outlines how to diagnose and fix a wrong answer that doesn't error. ## Four ways to be wrong A wrong answer has four common causes: | Cause | What you'd see | What you'd fix | | --- | --- | --- | | Retrieval found the wrong thing | The retrieval step returned a stale or irrelevant document | Your index, or the query you build | | The context was incomplete | Nothing in the trace holds the fact the answer needed | What you fetch, and when | | A tool returned junk | The tool call errored or came back empty, and the answer carried on | The tool, or how you handle its failure | | The model ignored what it was given | Everything upstream is correct and the answer still isn't | The prompt, or the model | ## How the trace tells them apart A [trace](/docs/ai-observability/traces.md) holds the whole interaction in order: what the user asked, every retrieval and tool call along the way, and what the model produced. Each step is its own event: - A [span](/docs/ai-observability/spans.md) for the work around the model - A [generation](/docs/ai-observability/generations.md) for the call itself You can read the trace top to bottom to see what the model knew at each point. In Fig. 1, the top hit from retrieval is `auth-v1`. The model was handed the old authentication page and answered from it faithfully. That's the first cause, and you can only see it because the span sits above the generation in the same trace. $ai\_traceanswer\_docs\_questionperson 8815 · 6.1s · $0.014$ai\_spanretrieve\_docs0.4s · 3 docs · top hit: auth-v1$ai\_generationanswer\_question2,900 in · 180 out · no error Fig. 1 – Retrieval handed the model the old header. ## What else is attached Because AIO events are like other PostHog events, a trace holds the rest of the story too: - The person who triggered it, and everything else they did that day. - Their [session replay](/docs/session-replay.md), so you can watch what they did with the answer. - Any [exception](/docs/error-tracking.md) the request threw, on the same trace. So "a user says the bot was wrong" becomes the request, the document it was given, and the moment they gave up. ## Check whether it's a pattern A single trace is an anecdote. Before you ship a fix, find out whether it's systemic: - Filter for the same failure over the last week - Check the [errors tab](/docs/ai-observability/errors.md) if it threw one - See whether the same eval fails on the same shape of question If it is a pattern, it's worth automating. An AI observability [scout](/docs/self-driving/scouts.md) watches these signals on a schedule and files an investigated report, so the next one arrives already read. ## Find traces with PostHog AI This prompt pulls the traces most likely to be worth reading to the surface: - The ones from the last 24 hours that errored or failed an evaluation - Each with its person, model, input, output, and the eval's reasoning - Then what the worst five have in common, and which single cause explains the most of them Prompt for PostHog AI ```text Show me the LLM traces from the last 24 hours where $ai_is_error is true or an evaluation failed. For each one, give me the trace id, the person, the model, the input, the output, and the reasoning from any failed evaluation. Then tell me what the worst five have in common – same model, same prompt version, same tool, same kind of question – and which single cause would explain the most of them. If my project has no real $ai_generation traffic yet, just say so and tell me how to instrument AI observability using the wizard or manual install. ``` [Find these with PostHog AI](https://app.posthog.com/#panel=max:!Show%20me%20the%20LLM%20traces%20from%20the%20last%2024%20hours%20where%20%24ai_is_error%20is%20true%20or%20an%20evaluation%20failed.%20For%20each%20one%2C%20give%20me%20the%20trace%20id%2C%20the%20person%2C%20the%20model%2C%20the%20input%2C%20the%20output%2C%20and%20the%20reasoning%20from%20any%20failed%20evaluation.%20Then%20tell%20me%20what%20the%20worst%20five%20have%20in%20common%20%E2%80%93%20same%20model%2C%20same%20prompt%20version%2C%20same%20tool%2C%20same%20kind%20of%20question%20%E2%80%93%20and%20which%20single%20cause%20would%20explain%20the%20most%20of%20them.%20If%20my%20project%20has%20no%20real%20%24ai_generation%20traffic%20yet%2C%20just%20say%20so%20and%20tell%20me%20how%20to%20instrument%20AI%20observability%20using%20the%20wizard%20or%20manual%20install.)It pulls the traces and reads them back to you. It doesn't build anything. Needs AI Observability instrumented. ``` npx @posthog/wizard ai-observability ``` See also: [Traces](/docs/ai-observability/traces.md) · [Link to Session Replay](/docs/ai-observability/link-session-replay.md) · [Errors](/docs/ai-observability/errors.md) [‹ Measure whether users keep the answer](/pocket-guides/ai-observability/throw-away-answers.md)[All guides](/pocket-guides.md)p. 6 of 6 Debug a wrong answer with traces[Find these with PostHog AI](https://app.posthog.com/#panel=max:!Show%20me%20the%20LLM%20traces%20from%20the%20last%2024%20hours%20where%20%24ai_is_error%20is%20true%20or%20an%20evaluation%20failed.%20For%20each%20one%2C%20give%20me%20the%20trace%20id%2C%20the%20person%2C%20the%20model%2C%20the%20input%2C%20the%20output%2C%20and%20the%20reasoning%20from%20any%20failed%20evaluation.%20Then%20tell%20me%20what%20the%20worst%20five%20have%20in%20common%20%E2%80%93%20same%20model%2C%20same%20prompt%20version%2C%20same%20tool%2C%20same%20kind%20of%20question%20%E2%80%93%20and%20which%20single%20cause%20would%20explain%20the%20most%20of%20them.%20If%20my%20project%20has%20no%20real%20%24ai_generation%20traffic%20yet%2C%20just%20say%20so%20and%20tell%20me%20how%20to%20instrument%20AI%20observability%20using%20the%20wizard%20or%20manual%20install.)