> AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt # Review and compare experiments Canonical URL: https://posthog.com/docs/ai-evals/experiments > **Cloud region:** This page uses US Cloud hosts. For an EU Cloud project, use `https://eu.posthog.com` instead of `https://app.posthog.com` (app host for links). ## On this page - [Read scores alongside coverage](#read-scores-alongside-coverage) - [Edit experiment items and results](#edit-experiment-items-and-results) - [Use results in your CI](#use-results-in-your-ci) An experiment is one run of your test cases against your agent. After your CI or a local script uploads an experiment, compare it with earlier runs to find the change that made a score drop. To run your first experiment, see [Offline evals](https://posthog.com/docs/ai-evals/offline-evals.md#run-your-first-experiment). Open [**AI Evals** > **Offline evals** > **Experiments**](https://app.posthog.com/ai-evals/evaluations/offline/experiments): 1. Select **Completed** in the upload-state filter to exclude runs that are still uploading or failed before all results arrived. 2. Click **Choose scores** to select the scorers shown in **Score trends**. Open a scorer's history to inspect its versions and the experiments behind the trend. 3. Select **Support agent baseline**. Each row is an experiment item, and each scorer version has its own column. 4. Click **Fail** under **Correct arguments** for `refund_other_order`. The inspector shows why: the agent selected the right tool but passed order ID `1001` instead of `1004`. ![The refund case failed the argument check: the agent sent order ID 1001 instead of 1004. The inspector shows the input, output, expected output, and reasoning.](https://raw.githubusercontent.com/PostHog/pr-assets/1c77ff41c95df05b609ba99d0a273c6094d8c55a/2026/11/92f4b7e6-4735-4600-9016-06b5c4dc6569.jpg)1![The refund case failed the argument check: the agent sent order ID 1001 instead of 1004. The inspector shows the input, output, expected output, and reasoning.](https://raw.githubusercontent.com/PostHog/pr-assets/1c77ff41c95df05b609ba99d0a273c6094d8c55a/2026/11/38927b3b-c40d-47ac-bb9f-e64df4d228bf.jpg)1 - 1 Expected order 1004; received order 1001. To fix the demo, pass the extracted order ID to the cancellation and refund tools, and return `look_up_order` for delivery questions. Run the same cases again with the experiment name **Support agent routing fix**. Both scorers should pass all six cases. Compare runs with the same test cases and scorer versions so changes in the test don't hide changes in your agent. ## Read scores alongside coverage Pass rate includes only results with status `ok`. Eight passing results and two evaluator errors produce a 100% pass rate, with only eight scored items. Expand **Coverage** beside each scorer summary to check errors, skipped checks, and missing results. **Missing** means an uploaded item has no result for that scorer. It can't reveal cases you never uploaded, so compare the received item count with your expected count too. Composite calculations you haven't requested show **Not requested**. Numeric scorers show their mean. Numeric and categorical scorers need a passing rule to show a pass rate. Repeated trials each count as an item, so keep trial counts consistent when comparing runs. ## Edit experiment items and results Each reported result belongs to one item in one experiment and uses a specific scorer version. You can edit the item or its results even after the run finishes. When you change a reported value or status, PostHog recalculates dependent composite scores for that item. On the experiment handle, use the ID from the upload receipt to read the current revision. Call `get_item_revision()` / `getItemRevision()` for an item or `get_result_revision()` / `getResultRevision()` for a result. Pass that ID and revision with the edit or deletion. | Action | Python | TypeScript | | --- | --- | --- | | Fix the case key, trial, model version, or trace ID | `update_item` | `updateItem` | | Replace item input, output, expected output, or metadata | `update_item_payload` | `updateItemPayload` | | Remove the item payload and keep its scores | `delete_item_payload` | `deleteItemPayload` | | Delete the item and all its scores and payloads | `delete_item` | `deleteItem` | | Correct a reported result's status, value, or error code | `update_result` | `updateResult` | | Replace result reasoning, error message, or metadata | `update_result_payload` | `updateResultPayload` | | Remove the result payload and keep the score | `delete_result_payload` | `deleteResultPayload` | | Delete the result, its payload, and dependent composite scores | `delete_result` | `deleteResult` | If you change the input or either output, rerun the affected checks and update their scores. A `revision_conflict` means the record changed. Review it before retrying with the latest revision. Use a [personal API key](https://posthog.com/docs/api/personal-api-keys.md) with `evaluation:read` to read revisions and `evaluation:write` plus evaluation editor access to make changes. Working with results or deleting an entire item also requires `llm_analytics:read` and access to the affected scorers and composite inputs. ## Use results in your CI Use a [project secret API key](https://posthog.com/docs/api/project-secret-api-keys.md) with the `offline_evaluation_ingestion:write` scope for your CI uploads. It can create experiments, upload results, and mark runs complete or failed without depending on a teammate's account. Project secret keys can't create or read scorers or fetch datasets. Set up scorers and load any PostHog dataset with a personal key first. In your CI script, pass the project key as `secret_key` / `secretKey` and use saved scorer version IDs instead of calling `scorers.get()`. For a composite, save its version ID and the exact input versions it pins. Send those input results with each calculation request. In your CI, set `run_source="ci"` in Python or `runSource: "ci"` in TypeScript. Record your agent's commit SHA in `application_version` or `applicationVersion`. You can also record `model_version` and `prompt_version` (`modelVersion` and `promptVersion` in TypeScript). For runs that use multiple models, set `model_version` / `modelVersion` on each `EvaluationItem` to override the experiment's model. This requires `posthog` 7.76.0+ or `@posthog/ai` 8.17.0+. Completing an experiment means the run finished, not that every case passed. To block a deployment, check scores against your own threshold in the test script and exit with a failure code.