- Is this surface working? Look at accepted ÷ shown, per feature. A rate that's fine on one surface and poor on another might signal a product problem, not necessarily a model one.
- Which model wins on your work? Split acceptance by
$ai_model. Benchmarks tell you which model wins on someone else's test set – this tells you which one wins on your users'. Ship the change behind a feature flag and read acceptance for each variant. - Wrong, or just unwanted? Read acceptance next to the eval pass rate from Score your AI's answers with evals. A high pass rate with low acceptance means the answers are correct and nobody wants them which is a different fix entirely.
Measure whether users keep the answer
You can have an AI feature that's fast, cheap, stays on-topic, and passes its evals – but users still regenerate outputs consistently. Every regeneration is a user telling you the first answer wasn't good enough.
Typical monitoring won't show why because the call succeeded and the cost was normal, nothing errored.
When you monitor for this scenario, Fig. 1 is what that looks like.
new model
~2,100 answers a day shown and discarded – no error, no complaint
Answers shownAnswers accepted · 84% → 55%
How PostHog connects the two
The signal already exists in your product as a "regenerate" click or a copy button, but they probably aren't attached to the LLM call that produced the answer.
Every LLM event carries a $ai_trace_id. Put that same ID on your own events as a custom property, and each product event joins back to the generation behind it, along with its model, prompt version, and cost.
Which events to send
There are four moments that follow a generation worth tracking:
| Event | Fires when | What it tells you |
|---|---|---|
| Shown | The answer reaches the user | Your denominator – not every generation gets seen |
| Accepted | The user copies, sends, applies, or keeps it | The answer did its job |
| Regenerated | The user asks for another one | Rejection, counted |
| Edited | The user changes it before using it | Close, but not right |
If you already have events for these moments, use them. If not, create new ones. The names of the events don't matter, just be sure to add the $ai_trace_id custom property to each one.
What the numbers tell you
With those events flowing, you can start answering valuable questions:
Thumbs up and down interactions through user feedback adds an explicit rating on top, currently in beta. Acceptance costs nothing extra and doesn't ask the user to stop and rate anything.
Build it with PostHog AI
This prompt builds three things against your own data:
- The funnel from generation to shown to accepted, joined on
$ai_trace_id - That funnel broken down by model and feature
- And a trend for how often people regenerate.
Rename the events to match yours before you send it: