> AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt # Offline evaluations – AI Observability pocket guide Canonical URL: https://posthog.com/learn/ai-observability/pocket-guide/offline-evaluations > **Cloud region:** This page uses US Cloud hosts. For an EU Cloud project, use `https://eu.posthog.com` instead of `https://app.posthog.com` (app host for links). ## On this page - [Save the request and expected outcome](#save-the-request-and-expected-outcome) - [Change the instructions and compare the answers](#change-the-instructions-and-compare-the-answers) - [Run the evaluations before release](#run-the-evaluations-before-release) - [Prepare a test in your project](#prepare-a-test-in-your-project) Chapter 03 # Offline evaluations – AI Observability pocket guide Before releasing the support agent, Twig's engineers had tested an ordinary cancellation: a visitor wants a full refund one day before arrival. The support agent explained the seven-day notice rule, and the policy evaluation passed. That answer is correct. But the heating complaint reveals a gap in the engineers' testing: a broken essential amenity that qualifies for an exception to the cancellation policy. Twig's engineers need to fix the support agent's refund refusal without breaking the standard cancellation policy answer. They use [offline evaluations](https://posthog.com/docs/ai-evals/offline-evals.md) to run saved requests through the support agent and evaluate its answers before releasing the change. ## Save the request and expected outcome Twig's engineers create a [dataset](https://posthog.com/docs/ai-evals/datasets.md), a collection of test cases they can rerun. In PostHog, they open the recorded heating conversation and add it to the dataset to save it as a test case alongside the ordinary cancellation request. For the heating complaint, they save two different things: - `source_output` contains the support agent's original refusal. - `expected_output` describes what the support agent should do: apply the broken-amenity exception and offer relocation or a refund. The refusal is evidence of the problem, not the answer the engineers *want* the evaluation to reward. Switch between the cancellation and broken-amenity cases to compare their answers with the expected outcomes: The cancellation case passes because the visitor changed their mind too late to qualify for a full refund. The heating case fails because the support agent refuses a refund that Twig's policy permits. The engineers save the policy information with each case too. Otherwise, someone rerunning the heating complaint might give the support agent a different policy and end up testing a different situation. ## Change the instructions and compare the answers Twig's engineers update the support prompt to tell the agent to apply the broken-amenity exception before the ordinary cancellation window. They keep the visitor's request, policy, model, and evaluation criteria the same. They're trying to find out whether changing this instruction fixes the refusal. The original prompt is their [baseline](https://posthog.com/docs/ai-evals/offline-evals.md): the version they compare the updated prompt against. They can open the captured generation in PostHog's [playground](https://posthog.com/docs/ai-observability/playground.md), a workspace for trying prompt and model changes against recorded inputs, to inspect a candidate answer. But one better answer isn't enough. The engineers also run both saved cases through the support agent. Send the heating complaint with the updated instructions, then inspect the answer and evaluation results: The support agent now explains that the heating qualifies for the broken-amenity exception because it can't be repaired. It offers relocation or a refund, and the policy evaluation passes. The ordinary cancellation still passes too. The new instructions haven't made the support agent offer a refund whenever someone cancels at the last minute. The code evaluation passes for both heating answers because both mention `"7 days"`. Sentiment remains negative because the visitor's complaint hasn't changed. The policy evaluation (LLM judge) is the one that tells Twig's engineers whether this fix worked. ## Run the evaluations before release Twig's engineers automate both saved cases with a [test runner](https://posthog.com/docs/ai-evals/offline-evals.md). It runs the requests through the support agent and its tools, evaluates the answers with the LLM judge, and reports the results to PostHog through the SDK. They run it locally while editing the prompt, then in continuous integration (CI), the automated tests that run with each application change. PostHog records each run as an [experiment](https://posthog.com/docs/ai-evals/offline-evals.md). To compare the original and updated prompts fairly, the engineers keep the same: - [Dataset revision](https://posthog.com/docs/ai-evals/datasets.md), a fixed version of the saved cases. - Model, policy, and LLM judge criteria. - [Scorer](https://posthog.com/docs/ai-evals/offline-evals.md), which defines the result's type and passing rule. Their “Policy correctness” scorer records `true` as a pass and `false` as a failure. The runner performs the evaluation; the scorer tells PostHog how to interpret the reported result. The cancellation case is a [regression test](https://posthog.com/docs/ai-evals/offline-evals.md): it catches the heating fix breaking an answer that already worked. The engineers add more cases as new failures appear. Errors and `N/A` results stay separate from passes. If the runner couldn't load the policy, it hasn't verified the answer. ## Prepare a test in your project Copy this prompt into [PostHog AI](https://app.posthog.com/#panel=max:Help%20me%20prepare%20offline%20evaluations%20for%20a%20prompt%20change%20in%20my%20project.%20Find%20one%20confirmed%20failure%20and%20one%20successful%20request%20that%20should%20keep%20working.%20Ask%20me%20to%20confirm%20the%20expected%20outcome%20for%20each%2C%20keeping%20the%20captured%20answer%20separate.%0A%0AIdentify%20the%20messages%2C%20tool%20results%2C%20and%20reference%20facts%20needed%20to%20reproduce%20both%20cases.%20Propose%20one%20prompt%20change%2C%20then%20explain%20how%20to%20compare%20the%20original%20and%20updated%20prompts%20using%20the%20same%20dataset%20revision%2C%20model%2C%20and%20evaluation%20criteria.%0A%0AShow%20what%20my%20local%20or%20CI%20runner%20must%20execute%20and%20how%20to%20report%20results%20to%20PostHog.%20Keep%20evaluation%20errors%20and%20N%2FA%20results%20separate%20from%20passes.%20Explain%20how%20to%20verify%20which%20prompt%20version%20the%20application%20used.%20Don't%20run%20the%20experiment%20or%20change%20the%20production%20prompt%20without%20my%20confirmation.): ```text Help me prepare offline evaluations for a prompt change in my project. Find one confirmed failure and one successful request that should keep working. Ask me to confirm the expected outcome for each, keeping the captured answer separate. Identify the messages, tool results, and reference facts needed to reproduce both cases. Propose one prompt change, then explain how to compare the original and updated prompts using the same dataset revision, model, and evaluation criteria. Show what my local or CI runner must execute and how to report results to PostHog. Keep evaluation errors and N/A results separate from passes. Explain how to verify which prompt version the application used. Don't run the experiment or change the production prompt without my confirmation. ``` [Prompts →](https://posthog.com/learn/ai-observability/pocket-guide/prompts.md) `npx @posthog/wizard ai-observability` **Explore further** [Datasets](https://posthog.com/docs/ai-evals/datasets.md) · [Offline evaluations](https://posthog.com/docs/ai-evals/offline-evals.md) · [Prompt management](https://posthog.com/docs/prompt-management.md) · [Playground](https://posthog.com/docs/ai-observability/playground.md)