> AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt

# Measure whether users keep the answer – AI Observability pocket guide

[](/pocket-guides.md)Aa

[](/pocket-guides/ai-observability/ai-cost.md)[](/pocket-guides/ai-observability/unexplained-answers.md)

# Measure whether users keep the answer – AI Observability pocket guide

You can have an AI feature that's fast, cheap, stays on-topic, and passes its evals – but users still regenerate outputs consistently. Every regeneration is a user telling you the first answer wasn't good enough.

Typical monitoring won't show why because the call succeeded and the cost was normal, nothing errored.

When you monitor for this scenario, Fig. 1 is what that looks like.

new model

~2,100 answers a day shown and discarded – no error, no complaint

Answers shownAnswers accepted · 84% → 55%

Fig. 1 – Every answer arrived and monitoring looked normal, but two in five got thrown away.

## How PostHog connects the two

The signal already exists in your product as a "regenerate" click or a copy button, but they probably aren't attached to the LLM call that produced the answer.

Every LLM event carries a `$ai_trace_id`. Put that same ID on your own events as a [custom property](/docs/ai-observability/custom-properties.md), and each product event joins back to the [generation](/docs/ai-observability/generations.md) behind it, along with its model, prompt version, and cost.

## Which events to send

There are four moments that follow a generation worth tracking:

| Event | Fires when | What it tells you |
| --- | --- | --- |
| Shown | The answer reaches the user | Your denominator – not every generation gets seen |
| Accepted | The user copies, sends, applies, or keeps it | The answer did its job |
| Regenerated | The user asks for another one | Rejection, counted |
| Edited | The user changes it before using it | Close, but not right |

If you already have events for these moments, use them. If not, create new ones. The names of the events don't matter, just be sure to add the `$ai_trace_id` custom property to each one.

## What the numbers tell you

With those events flowing, you can start answering valuable questions:

-   **Is this surface working?** Look at accepted ÷ shown, per feature. A rate that's fine on one surface and poor on another might signal a product problem, not necessarily a model one.
-   **Which model wins on your work?** Split acceptance by `$ai_model`. Benchmarks tell you which model wins on someone else's test set – this tells you which one wins on your users'. Ship the change behind a [feature flag](/docs/feature-flags.md) and read acceptance for each variant.
-   **Wrong, or just unwanted?** Read acceptance next to the eval pass rate from [Score your AI's answers with evals](/pocket-guides/ai-observability/quality-output.md). A high pass rate with low acceptance means the answers are correct and nobody wants them which is a different fix entirely.

Thumbs up and down interactions through [user feedback](/docs/ai-observability/collect-user-feedback.md) adds an explicit rating on top, currently in beta. Acceptance costs nothing extra and doesn't ask the user to stop and rate anything.

## Build it with PostHog AI

This prompt builds three things against your own data:

-   The funnel from generation to shown to accepted, joined on `$ai_trace_id`
-   That funnel broken down by model and feature
-   And a trend for how often people regenerate.

Rename the events to match yours before you send it:

Prompt for PostHog AI

```text
Build me an AI outcome funnel in PostHog. Step 1 is the $ai_generation event, step 2 is my ai_suggestion_shown event, step 3 is my ai_suggestion_accepted event, all joined on the $ai_trace_id property. Break the conversion down by $ai_model and by the ai_product property, and add a second insight showing the rate of ai_suggestion_regenerated per trace over time. Then tell me which model has the best acceptance rate once volume is accounted for. If my project doesn't have these events yet, build the full funnel anyway and tell me how to instrument AI observability using the wizard or manual install.
```

[Build this with PostHog AI](https://app.posthog.com/#panel=max:!Build%20me%20an%20AI%20outcome%20funnel%20in%20PostHog.%20Step%201%20is%20the%20%24ai_generation%20event%2C%20step%202%20is%20my%20ai_suggestion_shown%20event%2C%20step%203%20is%20my%20ai_suggestion_accepted%20event%2C%20all%20joined%20on%20the%20%24ai_trace_id%20property.%20Break%20the%20conversion%20down%20by%20%24ai_model%20and%20by%20the%20ai_product%20property%2C%20and%20add%20a%20second%20insight%20showing%20the%20rate%20of%20ai_suggestion_regenerated%20per%20trace%20over%20time.%20Then%20tell%20me%20which%20model%20has%20the%20best%20acceptance%20rate%20once%20volume%20is%20accounted%20for.%20If%20my%20project%20doesn't%20have%20these%20events%20yet%2C%20build%20the%20full%20funnel%20anyway%20and%20tell%20me%20how%20to%20instrument%20AI%20observability%20using%20the%20wizard%20or%20manual%20install.)Opens PostHog AI with the prompt ready. Rename the events to match your own.

See also: [Custom properties](/docs/ai-observability/custom-properties.md) · [Collecting user feedback](/docs/ai-observability/collect-user-feedback.md) · [Funnels](/docs/product-analytics/funnels.md)

[‹ Attribute AI cost to users and features](/pocket-guides/ai-observability/ai-cost.md)[All guides](/pocket-guides.md)p. 5 of 6[Debug a wrong answer with traces ›](/pocket-guides/ai-observability/unexplained-answers.md)

Measure whether users keep the answer[Build this with PostHog AI](https://app.posthog.com/#panel=max:!Build%20me%20an%20AI%20outcome%20funnel%20in%20PostHog.%20Step%201%20is%20the%20%24ai_generation%20event%2C%20step%202%20is%20my%20ai_suggestion_shown%20event%2C%20step%203%20is%20my%20ai_suggestion_accepted%20event%2C%20all%20joined%20on%20the%20%24ai_trace_id%20property.%20Break%20the%20conversion%20down%20by%20%24ai_model%20and%20by%20the%20ai_product%20property%2C%20and%20add%20a%20second%20insight%20showing%20the%20rate%20of%20ai_suggestion_regenerated%20per%20trace%20over%20time.%20Then%20tell%20me%20which%20model%20has%20the%20best%20acceptance%20rate%20once%20volume%20is%20accounted%20for.%20If%20my%20project%20doesn't%20have%20these%20events%20yet%2C%20build%20the%20full%20funnel%20anyway%20and%20tell%20me%20how%20to%20instrument%20AI%20observability%20using%20the%20wizard%20or%20manual%20install.)