AI evaluations, explained side by side.
Offline evals test your changes on saved cases before they go live. Online evals grade the real thing as it happens." Pick a path below and see what each one catches.
Offline evals
Test known casesLike unit / integration tests for your AI. Run a test suite on your laptop or in your CI.
Offline · Input
user: “Can I return an item I bought 45 days ago?”
This is a test case you got from a real user input. It’s defined in your code and loaded by your test runner.
Online evals
Discover missed casesReal users go beyond your finite test suite. Monitor their interactions in PostHog.
Online · Input
user: “This was a gift, I have no receipt, and it broke after 45 days. I’m frustrated. Can you replace it?”
This is an unexpected request from a real user in prod. This user is asking about a defective product, but your test suite only covers late returns.
Offline · Output
Waiting for a new test run…
Online · Output
Waiting for the app’s response…
Offline · Evaluator
Does the answer apply the right policy to this request?
Scores the answer using the input, output, tool call, and reference answer.
Online · Evaluator
Does the answer apply the right policy to this request?
Scores the answer using the input, output, and tool call. Production requests have no reference answer.
4Results in PostHog
Same destination, different workflows.
Offline evals
Not yetWaiting for your runner to report its results.
Online evals
Not yetWaiting for the evaluation results.