> AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt # Scorers Canonical URL: https://posthog.com/docs/ai-evals/scorers > **Cloud region:** This page uses US Cloud hosts. For an EU Cloud project, use `https://eu.posthog.com` instead of `https://app.posthog.com` (app host for links). A scorer defines what a result means: its name, value type, and passing rule. A reported scorer records scores that your evaluation code calculates. A composite scorer calculates a score in PostHog from other scores on the same item. For the basics, see [how offline evals work](https://posthog.com/docs/ai-evals/offline-evals.md#how-offline-evals-work). ## Add more scores Use a separate scorer for each thing you want to measure, such as correctness, relevance, or whether the agent selected the right tool. | Scorer type | Result value | Example | | --- | --- | --- | | Boolean | `true` or `false` | Did the agent complete the task correctly? | | Numeric | A number | How relevant is the answer, from zero to five? | | Categorical | A list of category keys | Which failure types occurred, such as `["wrong_tool"]`? | Create scorers with the SDK or in [**Scorers**](https://app.posthog.com/ai-evals/evaluations/scorers). Each result references a fixed scorer version, so configuration changes don't reinterpret earlier results. For each additional reported score, call `upload_result()` / `uploadResult()` with the same `EvaluationItem` and the new scorer's version ID. Include all reported and composite results in your expected result count. Use `reasoning` to explain a score. Set `application_trace_id` / `applicationTraceId` on the item to link it to an [AI Observability trace](https://posthog.com/docs/ai-observability/traces.md). ### Failed checks and evaluator errors Use `value=False` / `value: false` for a failed check. The default status, `ok`, means the evaluator produced a score, not that the item passed. If the evaluator can't produce a score, upload `status="error"` / `status: "error"`, omit `value`, and include an `error_message` / `errorMessage`. Use `skipped` for a check you didn't run, or `not_applicable` when a scorer doesn't apply to an item. ## Combine scores with a composite scorer A composite scorer combines reported scores for one item into a boolean, numeric, or categorical result. For the support-agent example, **Support task passed** requires both **Correct tool** and **Correct arguments** to pass. Three of the six cases pass both checks, giving a 50% pass rate. ### Create the composite Use the latest SDK. Save this as `create_composite.py` or `create_composite.mts` and run it once with the quickstart's environment variables and existing scorers. ### Python ```python import os from posthog.ai.evaluations import OfflineEvaluations, ScorerComposition with OfflineEvaluations( project_id=int(os.environ["POSTHOG_PROJECT_ID"]), secret_key=os.environ["POSTHOG_PERSONAL_API_KEY"], host=os.environ["POSTHOG_HOST"], ) as evaluations: tool = evaluations.scorers.get(os.environ["POSTHOG_TOOL_SCORER_ID"]) arguments = evaluations.scorers.get(os.environ["POSTHOG_ARGUMENTS_SCORER_ID"]) composition: ScorerComposition = { "expression": "tool_correct and arguments_correct", "inputs": [ {"alias": "tool_correct", "scorer_version_id": tool.current_version_id}, { "alias": "arguments_correct", "scorer_version_id": arguments.current_version_id, }, ], } # Optional: check the formula without saving a scorer. evaluations.scorers.validate_composition(kind="boolean", composition=composition) composite = evaluations.scorers.create( name="Support task passed", kind="boolean", calculation="composite", config={"true_is_failure": False}, composition=composition, ) print(composite.id) ``` ### TypeScript ```typescript import { OfflineEvaluations } from "@posthog/ai/evaluations" const evaluations = new OfflineEvaluations({ projectId: Number(process.env.POSTHOG_PROJECT_ID), secretKey: process.env.POSTHOG_PERSONAL_API_KEY!, host: process.env.POSTHOG_HOST!, }) try { const tool = await evaluations.scorers.get(process.env.POSTHOG_TOOL_SCORER_ID!) const argumentsScorer = await evaluations.scorers.get( process.env.POSTHOG_ARGUMENTS_SCORER_ID! ) const composition = { expression: "tool_correct and arguments_correct", inputs: [ { alias: "tool_correct", scorerVersionId: tool.currentVersionId }, { alias: "arguments_correct", scorerVersionId: argumentsScorer.currentVersionId, }, ], } // Optional: check the formula without saving a scorer. await evaluations.scorers.validateComposition({ kind: "boolean", composition }) const composite = await evaluations.scorers.create({ name: "Support task passed", kind: "boolean", calculation: "composite", config: { trueIsFailure: false }, composition, }) console.log(composite.id) } finally { evaluations.close() } ``` Save the printed scorer ID: ```bash export POSTHOG_COMPOSITE_SCORER_ID="" ``` You can also create this scorer in [**Scorers**](https://app.posthog.com/ai-evals/evaluations/scorers). Choose **Composite formula** and **Boolean**, add the two input scorers, and set their **Formula name** fields to `tool_correct` and `arguments_correct`. Use `tool_correct and arguments_correct` as the formula and **True means pass** as the passing value. Each input uses a fixed scorer version. New input versions don't update the composite automatically. Changing its formula or inputs creates a new composite version; existing results stay unchanged. ### Request a calculation for each item Upload the reported inputs and a calculation request together for each item. Creating the scorer alone doesn't calculate results. In `support_eval.py` or `support_eval.mts`, add `EvaluationResult` to the SDK imports. Keep the agent, evaluator, and cases, then replace the client block below them with this code. It uses the input versions saved in the composite. ### Python ```python with OfflineEvaluations( project_id=int(os.environ["POSTHOG_PROJECT_ID"]), secret_key=os.environ["POSTHOG_PERSONAL_API_KEY"], host=os.environ["POSTHOG_HOST"], ) as evaluations: composite = evaluations.scorers.get(os.environ["POSTHOG_COMPOSITE_SCORER_ID"]) input_versions = { source.alias: source.scorer_version_id for source in composite.input_scorers } experiment = evaluations.create_experiment( name="Support agent with composite", suite_key="support-agent", run_source="local", expected_item_count=len(cases), expected_result_count=len(cases) * 3, ) try: for case in cases: output = run_agent(case["input"]) item = EvaluationItem( case_key=case["id"], input=case["input"], output=output, expected_output=case["expected"], ) scores = evaluate_answer(output, case["expected"]) receipt = experiment.upload_results( [ EvaluationResult( item=item, scorer_version_id=input_versions["tool_correct"], **scores["tool"], ), EvaluationResult( item=item, scorer_version_id=input_versions["arguments_correct"], **scores["arguments"], ), EvaluationResult( item=item, scorer_version_id=composite.current_version_id, calculate=True, ), ] ) for chunk in receipt.chunks: for outcome in chunk.results: if outcome.status == "error": print( f"{case['id']}: {outcome.scorer_version_id}: {outcome.error_code}" ) except Exception: experiment.fail() raise experiment.complete() print(f"Completed experiment: {experiment.id}") ``` ### TypeScript ```typescript const evaluations = new OfflineEvaluations({ projectId: Number(process.env.POSTHOG_PROJECT_ID), secretKey: process.env.POSTHOG_PERSONAL_API_KEY!, host: process.env.POSTHOG_HOST!, }) try { const composite = await evaluations.scorers.get( process.env.POSTHOG_COMPOSITE_SCORER_ID! ) const inputVersions = new Map( composite.inputScorers.map((source) => [source.alias, source.scorerVersionId]) ) const experiment = await evaluations.createExperiment({ name: "Support agent with composite", suiteKey: "support-agent", runSource: "local", expectedItemCount: cases.length, expectedResultCount: cases.length * 3, }) try { for (const testCase of cases) { const output = await runAgent(testCase.input) const item = new EvaluationItem({ caseKey: testCase.id, input: testCase.input, output, expectedOutput: testCase.expected, }) const scores = evaluateAnswer(output, testCase.expected) const receipt = await experiment.uploadResults([ new EvaluationResult({ item, scorerVersionId: inputVersions.get("tool_correct")!, ...scores.tool, }), new EvaluationResult({ item, scorerVersionId: inputVersions.get("arguments_correct")!, ...scores.arguments, }), new EvaluationResult({ item, scorerVersionId: composite.currentVersionId, calculate: true, }), ]) for (const chunk of receipt.chunks) { for (const outcome of chunk.results) { if (outcome.status === "error") { console.error( `${testCase.id}: ${outcome.scorerVersionId}: ${outcome.errorCode}` ) } } } } } catch (error) { await experiment.fail() throw error } await experiment.complete() console.log(`Completed experiment: ${experiment.id}`) } finally { evaluations.close() } ``` For a composite result, send only the item, scorer version, and `calculate=True` / `calculate: true`. PostHog calculates the value. Check each returned `status` and `error_code` / `errorCode`: an upload can succeed while a calculation fails. This run expects **18** results: twelve reported and six composite. Accepted errors and skipped results count toward the total. Start a new experiment to test another composite version; completed runs can't accept new results. Sending the inputs and calculation together works with the quickstart's permissions, including on retries. To reuse previously uploaded inputs, also grant `evaluation:read` and `llm_analytics:read`. For larger batches, keep each item's results together so the SDK can send them in one request. Open the experiment to see **Support task passed** alongside the two reported scorers. Items without a calculation request show **Not requested** and don't affect its pass rate. ### Write formulas These formulas use the two boolean inputs from the support-agent example: | Output | Formula | Meaning | | --- | --- | --- | | Boolean | `tool_correct and arguments_correct` | Pass only when both checks are true. | | Numeric | `(number(tool_correct) + number(arguments_correct)) / 2` | Return 0, 0.5, or 1 for each item. | | Categorical | `"pass" if tool_correct and arguments_correct else "fail"` | Return a category key configured on the composite. | Formulas read raw input values. `number(True)` is 1 and `number(False)` is 0, even if an input scorer treats `true` as a failure. For categorical inputs, test category keys with expressions such as `"helpful" in relevance`. In the UI, the formula editor suggests a formula from the output type and inputs' passing rules. Manual edits stay unchanged when you edit inputs. Click **Use default formula** to restore the suggestion. You can use arithmetic (`+`, `-`, `*`, `/`), comparisons, `and`, `or`, `not`, conditional expressions, and the `number`, `min`, `max`, and `abs` functions. Each formula can reference up to 16 reported scorer versions. Every selected input must appear in the formula. Composite scorers can't use other composites as inputs or combine results across different items. ### Handle missing inputs and calculation errors If a pinned input result is missing, PostHog rejects the entire upload request. Include that input for the same item and version when you retry. If an input has status `error`, `skipped`, or `not_applicable`, the composite inherits that status without a value. If several statuses apply, the order is `error`, then `skipped`, then `not_applicable`. Input errors use the code `composite_input_error`. A calculation error, such as division by zero or a value outside the composite's configured bounds, produces an `error` result with code `composite_formula_error`. Valid reported inputs in the same request are still saved. These errors don't enter the composite's mean or pass rate.