AI observability

For instructions on how to authenticate to use this endpoint, see API overview.

Endpoints

GET
POST
GET
GET
GET
GET
GET
GET
GET
GET
GET
POST
POST

Retrieve ai observability traces evaluations

A trace's evaluation results, newest first and at most 250, each with its outcome and label ready to render, plus a roll-up of the latest run of each evaluation on each node.

Required API key scopes

llm_analytics:read

Path parameters

  • id
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/evaluations
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/evaluations/

Example response

Status 200
RESPONSE
{
"results": [
{
"id": "string",
"evaluationId": "string",
"evaluationName": "string",
"targetNodeId": "string",
"outcome": "pass",
"label": "string",
"score": 0,
"reasoning": "string",
"timestamp": "2019-08-24T14:15:22Z",
"isBackfill": true
}
],
"summary": {
"passed": 0,
"failed": 0,
"unrated": 0,
"nodes": [
{
"nodeId": "string",
"outcome": "pass"
}
]
}
}
Status 400 The encoded trace id is malformed.
Status 429 Too many queries are running for the project or organization.

Create ai observability traces exports

Starts writing the trace and every node, inputs and outputs included, to a JSON file. Poll the export until it completes, then download it.

Required API key scopes

llm_analytics:read

Path parameters

  • id
    string

Request parameters

  • timestamp_hint
    string

Response


Example request

POST /api/projects/:project_id/ai_observability/traces/:id/exports
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl
-H 'Content-Type: application/json'\
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/exports/\
-d timestamp_hint="string"

Example response

Status 201
RESPONSE
{
"id": 0,
"status": "pending",
"filename": "string",
"createdAt": "2019-08-24T14:15:22Z",
"expiresAfter": "2019-08-24T14:15:22Z",
"exception": "string"
}
Status 400 The timestamp hint or the encoded trace id is malformed.
Status 429 Too many exports were requested.

Retrieve ai observability traces exports

The state of one of your exports of this trace.

Required API key scopes

llm_analytics:read

Path parameters

  • export_id
    integer
  • id
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/exports/:export_id
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/exports/:export_id/

Example response

Status 200
RESPONSE
{
"id": 0,
"status": "pending",
"filename": "string",
"createdAt": "2019-08-24T14:15:22Z",
"expiresAfter": "2019-08-24T14:15:22Z",
"exception": "string"
}
Status 400 The encoded trace id is malformed.
Status 404 You have no export with this id for this trace.

Retrieve ai observability traces exports content

Downloads a completed export as a JSON file.

Required API key scopes

llm_analytics:read

Path parameters

  • export_id
    integer
  • id
    string

Query parameters

  • purpose
    string
    Default: download
    One of: "download""copy"

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/exports/:export_id/content
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/exports/:export_id/content/

Example response

Status 200
RESPONSE
{
"trace": {
"id": "string",
"name": "string",
"createdAt": "2019-08-24T14:15:22Z",
"sessionId": "string",
"person": {
"distinctId": "string",
"label": "string"
},
"totals": {
"costUsd": 0,
"inputTokens": 0,
"outputTokens": 0,
"cacheReadTokens": 0,
"cacheWriteTokens": 0,
"latencyMs": 0
},
"hasError": true,
"errorCount": 0,
"tree": [
{
"id": "string",
"kind": "trace",
"name": "string",
"model": "string",
"stats": {
"costUsd": 0,
"inputTokens": 0,
"outputTokens": 0,
"cacheReadTokens": 0,
"cacheWriteTokens": 0,
"latencyMs": 0
},
"hasError": true,
"children": [
{}
]
}
],
"timeline": [
{
"id": "string",
"kind": "trace",
"name": "string",
"depth": 0,
"startMs": 0,
"durationMs": 0,
"hasError": true
}
],
"totalMs": 0,
"threadNodeIds": [
"string"
],
"defaultNodeId": "string",
"nodeIdsByAlias": {
"property1": "string",
"property2": "string"
},
"git": {
"branch": "string",
"repo": "string"
}
},
"truncated": true,
"nodes": [
{
"id": "string",
"kind": "trace",
"event": {
"name": "string",
"uuid": "string",
"timestamp": "2019-08-24T14:15:22Z",
"properties": {},
"ioKeys": {
"input": "string",
"output": "string",
"tools": "string"
}
}
}
]
}
Status 400 The encoded trace id or the purpose is malformed.
Status 404 You have no export with this id for this trace.
Status 409 The export is still being written, or failed.
RESPONSE
{
"detail": "string"
}

Retrieve ai observability traces feedback

End-user feedback on a trace: scores and comments from its feedback events, survey responses linked to it, and surveys shown but not answered.

Required API key scopes

llm_analytics:read

Path parameters

  • id
    string

Query parameters

  • timestamp_hint
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/feedback
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/feedback/

Example response

Status 200
RESPONSE
{
"scores": [
{
"id": "string",
"name": "string",
"value": "string",
"nodeId": "string",
"timestamp": "2019-08-24T14:15:22Z"
}
],
"responses": [
{
"kind": "text",
"id": "string",
"text": "string",
"nodeId": "string",
"timestamp": "2019-08-24T14:15:22Z"
}
],
"unanswered": [
{
"survey": {
"id": "string",
"name": "string"
},
"timestamp": "2019-08-24T14:15:22Z"
}
],
"surveyResponsesStatus": "ok",
"summary": {
"total": 0,
"thumbsUp": 0,
"thumbsDown": 0
}
}
Status 400 The timestamp hint or the encoded trace id is malformed.
Status 429 Too many queries are running for the project or organization.

Retrieve ai observability traces nodes

One node of a trace: its input and output ready to render, its event name, uuid and timestamp, its error and its metadata. The raw event properties are on the raw read.

Required API key scopes

llm_analytics:read

Path parameters

  • id
    string
  • node_id
    string

Query parameters

  • timestamp_hint
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/nodes/:node_id
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/nodes/:node_id/

Example response

Status 200
RESPONSE
{
"id": "string",
"kind": "trace",
"error": "string",
"metadata": {
"provider": "string",
"temperature": 0,
"sessionId": "string",
"promptName": "string",
"promptVersion": 0
},
"event": {
"name": "string",
"uuid": "string",
"timestamp": "2019-08-24T14:15:22Z"
},
"content": {
"input": {
"kind": "messages",
"messages": [
{
"id": "string",
"role": "system",
"parts": [
{
"kind": "text",
"text": "string"
}
],
"isInternal": true,
"sourceNodeId": "string"
}
]
},
"output": {
"kind": "messages",
"messages": [
{
"id": "string",
"role": "system",
"parts": [
{
"kind": "text",
"text": "string"
}
],
"isInternal": true,
"sourceNodeId": "string"
}
]
}
}
}
Status 400 The timestamp hint or an encoded id is malformed.
Status 404 No node with this id in the trace.
Status 429 Too many queries are running for the project or organization.

Retrieve ai observability traces nodes raw

One node of a trace as ingested: its event properties and the keys that hold its input, output and tools, with nothing normalized.

Required API key scopes

llm_analytics:read

Path parameters

  • id
    string
  • node_id
    string

Query parameters

  • timestamp_hint
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/nodes/:node_id/raw
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/nodes/:node_id/raw/

Example response

Status 200
RESPONSE
{
"id": "string",
"kind": "trace",
"event": {
"name": "string",
"uuid": "string",
"timestamp": "2019-08-24T14:15:22Z",
"properties": {},
"ioKeys": {
"input": "string",
"output": "string",
"tools": "string"
}
}
}
Status 400 The timestamp hint or an encoded id is malformed.
Status 404 No node with this id in the trace.
Status 429 Too many queries are running for the project or organization.

Retrieve ai observability traces pull request

The pull request the trace's git branch belongs to, from engineering analytics.

Required API key scopes

llm_analytics:readengineering_analytics:read

Path parameters

  • id
    string

Query parameters

  • timestamp_hint
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/pull_request
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/pull_request/

Example response

Status 200
RESPONSE
{
"pullRequest": {
"repoOwner": "string",
"repoName": "string",
"number": 0,
"title": "string"
}
}
Status 400 The timestamp hint or the encoded trace id is malformed.
Status 403 The engineering-analytics feature flag is off for the project, or the API key lacks a scope.
Status 429 Too many queries are running for the project or organization.

Retrieve ai observability traces review

The trace's human review with its scores ready to render, and the review queue the trace waits in.

Required API key scopes

llm_analytics:read

Path parameters

  • id
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/review
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/review/

Example response

Status 200
RESPONSE
{
"review": {
"reviewedBy": "string",
"reviewedAt": "2019-08-24T14:15:22Z",
"comment": "string",
"scores": [
{
"name": "string",
"value": "string",
"label": "string"
}
]
},
"queue": {
"id": "string",
"name": "string",
"itemId": "string"
}
}
Status 400 The encoded trace id is malformed.


Retrieve ai observability traces summary

The trace's AI summary: pending while it is written, then ready with links to its steps, or failed with the reason. A summary is kept for an hour and shown only to users whose property access matches the requester's.

Required API key scopes

llm_analytics:read

Path parameters

  • id
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/summary
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/summary/

Example response

Status 200
RESPONSE
{
"status": "pending",
"content": {
"title": "string",
"bullets": [
{
"text": "string",
"nodeId": "string"
}
],
"notes": [
{
"text": "string",
"nodeId": "string"
}
],
"flow": "string",
"model": "string",
"generatedAt": "2019-08-24T14:15:22Z"
},
"failure": {
"reason": "trace_not_found",
"message": "string"
}
}
Status 400 The encoded trace id is malformed.
Status 403 The organization has not approved AI data processing.
Status 404 No summary of this trace that this user may see.

Create ai observability traces summary

Request an AI summary of the trace. A ready summary is returned as is unless regenerate is set; otherwise a job starts writing one, or the request joins the job already writing it.

Required API key scopes

llm_analytics:write

Path parameters

  • id
    string

Query parameters

  • timestamp_hint
    string

Request parameters

  • regenerate
    boolean
    Default: false

Response


Example request

POST /api/projects/:project_id/ai_observability/traces/:id/summary
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl
-H 'Content-Type: application/json'\
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/summary/\
-d regenerate="boolean"

Example response

Status 200 A ready or failed summary.
RESPONSE
{
"status": "pending",
"content": {
"title": "string",
"bullets": [
{
"text": "string",
"nodeId": "string"
}
],
"notes": [
{
"text": "string",
"nodeId": "string"
}
],
"flow": "string",
"model": "string",
"generatedAt": "2019-08-24T14:15:22Z"
},
"failure": {
"reason": "trace_not_found",
"message": "string"
}
}
Status 202 The summary is being written.
RESPONSE
{
"status": "pending",
"content": {
"title": "string",
"bullets": [
{
"text": "string",
"nodeId": "string"
}
],
"notes": [
{
"text": "string",
"nodeId": "string"
}
],
"flow": "string",
"model": "string",
"generatedAt": "2019-08-24T14:15:22Z"
},
"failure": {
"reason": "trace_not_found",
"message": "string"
}
}
Status 400 The timestamp hint or the encoded trace id is malformed.
Status 403 The organization has not approved AI data processing.
Status 429 Too many summaries requested.

Create ai observability traces availability

Required API key scopes

llm_analytics:read

Request parameters

  • trace_ids
    array

Response


Example request

POST /api/projects/:project_id/ai_observability/traces/availability
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl
-H 'Content-Type: application/json'\
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/availability/\
-d trace_ids="array"

Example response

Status 200
RESPONSE
{
"available_trace_ids": [
"string"
]
}
Status 429 Too many queries are running for the project or organization.