AI observability

For instructions on how to authenticate to use this endpoint, see API overview.

Endpoints

GET
PATCH
DELETE
GET
POST
GET
GET
GET
GET
POST
GET
GET
GET
GET
GET
POST
GET
GET
GET
GET

Retrieve ai observability offline experiments results payload

Required API key scopes

evaluation:readllm_analytics:read

Path parameters

  • id
    string
  • result_id
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/offline_experiments/:id/results/:result_id/payload
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/offline_experiments/:id/results/:result_id/payload/

Example response

Status 200
RESPONSE
{
"revision": 0,
"deleted_at": "2019-08-24T14:15:22Z",
"id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
"payload_state": "not_provided",
"payload_expires_at": "2019-08-24T14:15:22Z",
"available": true,
"data": {
"reasoning": "string",
"error_message": "string",
"metadata": {}
}
}
Status 400
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 401
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 403
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 404
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 429
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 503
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}

Update ai observability offline experiments results payload

Required API key scopes

evaluation:writellm_analytics:read

Path parameters

  • id
    string
  • result_id
    string

Request parameters

  • revision
    integer
  • payload

Response


Example request

PATCH /api/projects/:project_id/ai_observability/offline_experiments/:id/results/:result_id/payload
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl -X PATCH \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/offline_experiments/:id/results/:result_id/payload/\
-d revision="integer",\
-d payload=undefined

Example response

Status 200
RESPONSE
{
"revision": 0
}
Status 400
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 401
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 403
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 404
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 409
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 429
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 503
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}

Delete ai observability offline experiments results payload

Required API key scopes

evaluation:writellm_analytics:read

Path parameters

  • id
    string
  • result_id
    string

Query parameters

  • revision
    integer

Example request

DELETE /api/projects/:project_id/ai_observability/offline_experiments/:id/results/:result_id/payload
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl -X DELETE \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/offline_experiments/:id/results/:result_id/payload/

Example response

Status 204 No response body
Status 400
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 401
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 403
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 404
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 409
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 429
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 503
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}

List all ai observability offline experiments scorer summaries

Required API key scopes

evaluation:readllm_analytics:read

Path parameters

  • id
    string

Query parameters

  • cursor
    string
  • limit
    integer
    Default: 50
  • scorer_definition_id
    string
  • scorer_version_ids
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/offline_experiments/:id/scorer_summaries
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/offline_experiments/:id/scorer_summaries/

Example response

Status 200
RESPONSE
{
"next": "http://example.com",
"previous": "http://example.com",
"count": 0,
"next_cursor": "string",
"results": [
{
"not_requested_count": 0,
"scorer": {
"calculation": "reported",
"composition": {
"expression": "string",
"inputs": [
{
"alias": "string",
"scorer_version_id": "d07df8d1-3b1c-4e6f-8d6d-dd2c83c1a72e"
}
]
},
"id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
"definition_id": "91c1994f-b1db-4fef-840d-7d3ab2984871",
"version": 0,
"kind": "categorical",
"name": "string",
"short_name": "string",
"description": "string",
"archived": true,
"config": {
"options": [
{
"key": "string",
"label": "string"
}
],
"selection_mode": "single",
"min_selections": 1,
"max_selections": 1,
"passing_rule": {
"categories": [
"string"
]
}
}
},
"observed_item_count": 0,
"result_count": 0,
"status_counts": {
"ok": 0,
"error": 0,
"skipped": 0,
"not_applicable": 0
},
"missing_result_count": 0,
"distinct_case_count": 0,
"items_with_case_key_count": 0,
"items_without_case_key_count": 0,
"trial_item_count": 0,
"distinct_trial_count": 0,
"mean": 0,
"true_count": 0,
"false_count": 0,
"true_rate": 0,
"pass_count": 0,
"fail_count": 0,
"pass_rate": 0,
"categories": [
{
"key": "string",
"count": 0,
"rate": 0,
"label": "string"
}
]
}
]
}
Status 400
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 401
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 403
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 404
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 429
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 503
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}

Create ai observability offline experiments upload

Required API key scopes

offline_evaluation_ingestion:write

Path parameters

  • id
    string

Request parameters

  • items
    Click to open
    array
  • results
    Click to open
    array

Response


Example request

POST /api/projects/:project_id/ai_observability/offline_experiments/:id/upload
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl
-H 'Content-Type: application/json'\
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/offline_experiments/:id/upload/\
-d results="array"

Example response

Status 200
RESPONSE
{
"items": [
{
"id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
"created": true,
"accepted_at": "2019-08-24T14:15:22Z"
}
],
"results": [
{
"id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
"created": true,
"accepted_at": "2019-08-24T14:15:22Z",
"item_id": "4d8cd62e-a579-4dae-af8c-3172f96f8f7c",
"scorer_version_id": "d07df8d1-3b1c-4e6f-8d6d-dd2c83c1a72e",
"status": "ok",
"error_code": "string"
}
]
}
Status 400
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 401
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 402
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 403
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 404
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 409
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 413
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 429
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}

List all ai observability offline scorers history

Required API key scopes

evaluation:readllm_analytics:read

Path parameters

  • id
    string

Query parameters

  • application_version
    string
  • breakdown
    string
    One of: "model_version""prompt_version""application_version""suite_key""run_source"
  • created_by
    integer
  • created_by_is_null
    boolean
  • cursor
    string
  • dataset_identifier
    string
  • dataset_revision_identifier
    string
  • dataset_source
    string
  • date_from
    string
  • date_to
    string
  • limit
    integer
    Default: 50
  • model_version
    string
  • prompt_version
    string
  • run_source
    string
  • scorer_version_ids
    string
  • search
    string
  • statuses
    string
  • suite_key
    string
  • tag_match
    string
    Default: any
    One of: "any""all"
  • tags
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/offline_scorers/:id/history
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/offline_scorers/:id/history/

Example response

Status 200
RESPONSE
{
"next": "http://example.com",
"previous": "http://example.com",
"count": 0,
"next_cursor": "string",
"results": [
{
"breakdown_value": "string",
"experiment": {
"updated_at": "2019-08-24T14:15:22Z",
"closed_item_count": 0,
"created_by": {
"id": 0,
"uuid": "095be615-a8ad-4c33-8e9c-c7612fbf6c9f",
"distinct_id": "string",
"first_name": "string",
"last_name": "string",
"email": "user@example.com",
"is_email_verified": true,
"hedgehog_config": {},
"role_at_organization": "engineering"
},
"tags": [
"string"
],
"id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
"name": "string",
"run_source": "ci",
"status": "uploading",
"started_at": "2019-08-24T14:15:22Z",
"created_at": "2019-08-24T14:15:22Z",
"finished_at": "2019-08-24T14:15:22Z",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"visible_result_count": 0,
"visible_scorer_definition_count": 0,
"visible_scorer_version_count": 0,
"result_count_scope": "string",
"suite_key": "string",
"dataset_source": "string",
"dataset_identifier": "string",
"dataset_revision_identifier": "string",
"dataset_revision_id": "e7047d72-9b1c-4feb-9c11-cc3d637eeab5",
"application_version": "string",
"model_version": "string",
"prompt_version": "string"
},
"summary": {
"not_requested_count": 0,
"scorer": {
"calculation": "reported",
"composition": {
"expression": "string",
"inputs": [
{
"alias": "string",
"scorer_version_id": "d07df8d1-3b1c-4e6f-8d6d-dd2c83c1a72e"
}
]
},
"id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
"definition_id": "91c1994f-b1db-4fef-840d-7d3ab2984871",
"version": 0,
"kind": "categorical",
"name": "string",
"short_name": "string",
"description": "string",
"archived": true,
"config": {
"options": [
{
"key": "string",
"label": "string"
}
],
"selection_mode": "single",
"min_selections": 1,
"max_selections": 1,
"passing_rule": {
"categories": [
"string"
]
}
}
},
"observed_item_count": 0,
"result_count": 0,
"status_counts": {
"ok": 0,
"error": 0,
"skipped": 0,
"not_applicable": 0
},
"missing_result_count": 0,
"distinct_case_count": 0,
"items_with_case_key_count": 0,
"items_without_case_key_count": 0,
"trial_item_count": 0,
"distinct_trial_count": 0,
"mean": 0,
"true_count": 0,
"false_count": 0,
"true_rate": 0,
"pass_count": 0,
"fail_count": 0,
"pass_rate": 0,
"categories": [
{
"key": "string",
"count": 0,
"rate": 0,
"label": "string"
}
]
}
}
]
}
Status 400
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 401
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 403
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 404
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 429
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}
Status 503
RESPONSE
{
"type": "string",
"code": "string",
"detail": "string",
"attr": "string",
"expected_item_count": 0,
"expected_result_count": 0,
"accepted_item_count": 0,
"accepted_result_count": 0,
"errors": [
{
"code": "string",
"detail": "string",
"attr": "string"
}
]
}

List all ai observability traces

The project's traces in a date range, newest first, a page at a time. The first page also carries how many traces the selection holds.

Required API key scopes

llm_analytics:read

Query parameters

  • cursor
    string
  • date_from
    string
  • date_to
    string
  • filter_test_accounts
    boolean | null
  • group_key
    string
  • group_type_index
    integer
  • limit
    integer
    Default: 50
  • person_id
    string
  • properties
    string
  • search
    string
  • view
    string
    One of: "errors""feedback"

Response


Example request

GET /api/projects/:project_id/ai_observability/traces
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/

Example response

Status 200
RESPONSE
{
"items": [
{
"id": "string",
"name": "string",
"createdAt": "2019-08-24T14:15:22Z",
"sessionId": "string",
"totals": {
"costUsd": 0,
"inputTokens": 0,
"outputTokens": 0,
"cacheReadTokens": 0,
"cacheWriteTokens": 0,
"latencyMs": 0
},
"hasError": true,
"errorCount": 0,
"totalsPartial": true,
"defaultNodeId": "string",
"firstError": "string",
"toolsCalled": [
"string"
],
"isSupportTrace": true,
"review": {
"review": {
"reviewedBy": "string",
"reviewedAt": "2019-08-24T14:15:22Z",
"comment": "string",
"scores": [
{
"name": "string",
"value": "string",
"label": "string"
}
]
},
"queue": {
"id": "string",
"name": "string",
"itemId": "string"
}
}
}
],
"counts": {
"total": 0,
"errors": 0,
"feedback": 0
},
"nextCursor": "string"
}
Status 400 A parameter, the cursor, the date range or a property filter is invalid.
Status 403 The key lacks a required scope, or a property filter needs the query:read scope.
Status 429 Too many queries are running for the project or organization.

Retrieve ai observability traces clusters

The clusters that list this trace, from the newest scheduled run of each clustering job that lists it.

Required API key scopes

llm_analytics:readai_observability_clusters:read

Path parameters

  • id
    string

Query parameters

  • created_at
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/clusters
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/clusters/

Example response

Status 200
RESPONSE
{
"clusters": [
{
"runId": "string",
"runAt": "2019-08-24T14:15:22Z",
"jobId": "string",
"jobName": "string",
"clusterId": 0,
"title": "string",
"label": "string",
"description": "string",
"isOutlier": true
}
]
}
Status 400 The created_at parameter or the encoded trace id is malformed.
Status 429 Too many queries are running for the project or organization.

Retrieve ai observability traces evaluations

A trace's evaluation results, newest first and at most 250, each with its outcome and label ready to render, plus a roll-up of the latest run of each evaluation on each node.

Required API key scopes

llm_analytics:read

Path parameters

  • id
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/evaluations
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/evaluations/

Example response

Status 200
RESPONSE
{
"results": [
{
"id": "string",
"evaluationId": "string",
"evaluationName": "string",
"targetNodeId": "string",
"outcome": "pass",
"label": "string",
"score": 0,
"reasoning": "string",
"timestamp": "2019-08-24T14:15:22Z",
"isBackfill": true
}
],
"summary": {
"passed": 0,
"failed": 0,
"unrated": 0,
"nodes": [
{
"nodeId": "string",
"outcome": "pass"
}
]
}
}
Status 400 The encoded trace id is malformed.
Status 429 Too many queries are running for the project or organization.

Create ai observability traces exports

Starts writing the trace and every node, inputs and outputs included, to a JSON file. Poll the export until it completes, then download it.

Required API key scopes

llm_analytics:read

Path parameters

  • id
    string

Request parameters

  • timestamp_hint
    string

Response


Example request

POST /api/projects/:project_id/ai_observability/traces/:id/exports
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl
-H 'Content-Type: application/json'\
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/exports/\
-d timestamp_hint="string"

Example response

Status 201
RESPONSE
{
"id": 0,
"status": "pending",
"filename": "string",
"createdAt": "2019-08-24T14:15:22Z",
"expiresAfter": "2019-08-24T14:15:22Z",
"exception": "string"
}
Status 400 The timestamp hint or the encoded trace id is malformed.
Status 429 Too many exports were requested.

Retrieve ai observability traces exports

The state of one of your exports of this trace.

Required API key scopes

llm_analytics:read

Path parameters

  • export_id
    integer
  • id
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/exports/:export_id
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/exports/:export_id/

Example response

Status 200
RESPONSE
{
"id": 0,
"status": "pending",
"filename": "string",
"createdAt": "2019-08-24T14:15:22Z",
"expiresAfter": "2019-08-24T14:15:22Z",
"exception": "string"
}
Status 400 The encoded trace id is malformed.
Status 404 You have no export with this id for this trace.

Retrieve ai observability traces exports content

Downloads a completed export as a JSON file.

Required API key scopes

llm_analytics:read

Path parameters

  • export_id
    integer
  • id
    string

Query parameters

  • purpose
    string
    Default: download
    One of: "download""copy"

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/exports/:export_id/content
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/exports/:export_id/content/

Example response

Status 200
RESPONSE
{
"trace": {
"id": "string",
"name": "string",
"createdAt": "2019-08-24T14:15:22Z",
"sessionId": "string",
"totals": {
"costUsd": 0,
"inputTokens": 0,
"outputTokens": 0,
"cacheReadTokens": 0,
"cacheWriteTokens": 0,
"latencyMs": 0
},
"hasError": true,
"errorCount": 0,
"totalsPartial": true,
"person": {
"distinctId": "string",
"label": "string"
},
"tree": [
{
"id": "string",
"kind": "trace",
"name": "string",
"label": "string",
"model": "string",
"stats": {
"costUsd": 0,
"inputTokens": 0,
"outputTokens": 0,
"cacheReadTokens": 0,
"cacheWriteTokens": 0,
"latencyMs": 0
},
"hasError": true,
"children": [
{}
]
}
],
"timeline": [
{
"id": "string",
"kind": "trace",
"name": "string",
"depth": 0,
"startMs": 0,
"durationMs": 0,
"hasError": true
}
],
"totalMs": 0,
"threadNodeIds": [
"string"
],
"defaultNodeId": "string",
"nodeIdsByAlias": {
"property1": "string",
"property2": "string"
},
"git": {
"branch": "string",
"repo": "string"
}
},
"truncated": true,
"nodes": [
{
"id": "string",
"kind": "trace",
"event": {
"name": "string",
"uuid": "string",
"timestamp": "2019-08-24T14:15:22Z",
"properties": {},
"ioKeys": {
"input": "string",
"output": "string",
"tools": "string"
}
}
}
]
}
Status 400 The encoded trace id or the purpose is malformed.
Status 404 You have no export with this id for this trace.
Status 409 The export is still being written, or failed.
RESPONSE
{
"detail": "string"
}

Retrieve ai observability traces feedback

End-user feedback on a trace: scores and comments from its feedback events, survey responses linked to it, and surveys shown but not answered.

Required API key scopes

llm_analytics:read

Path parameters

  • id
    string

Query parameters

  • timestamp_hint
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/feedback
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/feedback/

Example response

Status 200
RESPONSE
{
"scores": [
{
"id": "string",
"name": "string",
"value": "string",
"nodeId": "string",
"timestamp": "2019-08-24T14:15:22Z"
}
],
"responses": [
{
"kind": "text",
"id": "string",
"text": "string",
"nodeId": "string",
"timestamp": "2019-08-24T14:15:22Z"
}
],
"unanswered": [
{
"survey": {
"id": "string",
"name": "string"
},
"timestamp": "2019-08-24T14:15:22Z"
}
],
"surveyResponsesStatus": "ok",
"summary": {
"total": 0,
"thumbsUp": 0,
"thumbsDown": 0
},
"steps": [
{
"nodeId": "string",
"scores": [
{
"id": "string",
"name": "string",
"value": "string",
"nodeId": "string",
"timestamp": "2019-08-24T14:15:22Z"
}
],
"responseCount": 0,
"total": 0
}
]
}
Status 400 The timestamp hint or the encoded trace id is malformed.
Status 429 Too many queries are running for the project or organization.

Retrieve ai observability traces nodes

One node of a trace: its input and output ready to render, its event name, uuid and timestamp, its error and its metadata. The raw event properties are on the raw read.

Required API key scopes

llm_analytics:read

Path parameters

  • id
    string
  • node_id
    string

Query parameters

  • timestamp_hint
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/nodes/:node_id
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/nodes/:node_id/

Example response

Status 200
RESPONSE
{
"id": "string",
"kind": "trace",
"error": "string",
"metadata": {
"provider": "string",
"temperature": 0,
"sessionId": "string",
"promptName": "string",
"promptVersion": 0
},
"event": {
"name": "string",
"uuid": "string",
"timestamp": "2019-08-24T14:15:22Z"
},
"content": {
"input": {
"kind": "messages",
"messages": [
{
"id": "string",
"role": "system",
"parts": [
{
"kind": "text",
"text": "string"
}
],
"isInternal": true,
"sourceNodeId": "string"
}
]
},
"output": {
"kind": "messages",
"messages": [
{
"id": "string",
"role": "system",
"parts": [
{
"kind": "text",
"text": "string"
}
],
"isInternal": true,
"sourceNodeId": "string"
}
]
}
}
}
Status 400 The timestamp hint or an encoded id is malformed.
Status 404 No node with this id in the trace.
Status 429 Too many queries are running for the project or organization.

Retrieve ai observability traces nodes dataset item

One node of a trace as a dataset item: its input and output as ingested, and its custom properties as metadata. The root node gives the trace's input and output state with no metadata.

Required API key scopes

llm_analytics:read

Path parameters

  • id
    string
  • node_id
    string

Query parameters

  • timestamp_hint
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/nodes/:node_id/dataset_item
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/nodes/:node_id/dataset_item/

Example response

Status 200
RESPONSE
{
"clientItemId": "string",
"sourceTraceId": "string",
"sourceEventId": "string",
"sourceTimestamp": "2019-08-24T14:15:22Z",
"input": {},
"sourceOutput": {},
"metadata": {}
}
Status 400 The timestamp hint is missing or malformed, or an encoded id is malformed.
Status 404 No node with this id in the trace.
Status 429 Too many queries are running for the project or organization.

Create ai observability traces nodes parser test

Tests a custom parser recipe against one node without saving it: the node's input and output as they would render with the parser saved, and whether it parses every side that the project's saved parsers leave unparsed.

Required API key scopes

llm_analytics:read

Path parameters

  • id
    string
  • node_id
    string

Query parameters

  • timestamp_hint
    string

Request parameters

  • source
    string

Response


Example request

POST /api/projects/:project_id/ai_observability/traces/:id/nodes/:node_id/parser_test
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl
-H 'Content-Type: application/json'\
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/nodes/:node_id/parser_test/\
-d source="string"

Example response

Status 200
RESPONSE
{
"content": {
"input": {
"kind": "messages",
"messages": [
{
"id": "string",
"role": "system",
"parts": [
{
"kind": "text",
"text": "string"
}
],
"isInternal": true,
"sourceNodeId": "string"
}
]
},
"output": {
"kind": "messages",
"messages": [
{
"id": "string",
"role": "system",
"parts": [
{
"kind": "text",
"text": "string"
}
],
"isInternal": true,
"sourceNodeId": "string"
}
]
}
},
"input": {
"neededFixing": true,
"fixed": true,
"broken": true,
"error": {
"source": "existing_recipes",
"message": "string"
}
},
"output": {
"neededFixing": true,
"fixed": true,
"broken": true,
"error": {
"source": "existing_recipes",
"message": "string"
}
},
"passed": true
}
Status 400 The parser does not compile, with the reason in `detail`, or the timestamp hint or an encoded id is malformed.
Status 404 No node with this id in the trace.
Status 429 Too many queries are running for the project or organization.

Retrieve ai observability traces nodes raw

One node of a trace as ingested: its event properties and the keys that hold its input, output and tools, with nothing normalized.

Required API key scopes

llm_analytics:read

Path parameters

  • id
    string
  • node_id
    string

Query parameters

  • timestamp_hint
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/nodes/:node_id/raw
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/nodes/:node_id/raw/

Example response

Status 200
RESPONSE
{
"id": "string",
"kind": "trace",
"event": {
"name": "string",
"uuid": "string",
"timestamp": "2019-08-24T14:15:22Z",
"properties": {},
"ioKeys": {
"input": "string",
"output": "string",
"tools": "string"
}
}
}
Status 400 The timestamp hint or an encoded id is malformed.
Status 404 No node with this id in the trace.
Status 429 Too many queries are running for the project or organization.

Retrieve ai observability traces pull request

The pull request the trace's git branch belongs to, from engineering analytics.

Required API key scopes

llm_analytics:readengineering_analytics:read

Path parameters

  • id
    string

Query parameters

  • timestamp_hint
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/pull_request
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/pull_request/

Example response

Status 200
RESPONSE
{
"pullRequest": {
"repoOwner": "string",
"repoName": "string",
"number": 0,
"title": "string"
}
}
Status 400 The timestamp hint or the encoded trace id is malformed.
Status 403 The engineering-analytics feature flag is off for the project, or the API key lacks a scope.
Status 429 Too many queries are running for the project or organization.

Retrieve ai observability traces review

The trace's human review with its scores ready to render, and the review queue the trace waits in.

Required API key scopes

llm_analytics:read

Path parameters

  • id
    string

Response


Example request

GET /api/projects/:project_id/ai_observability/traces/:id/review
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
-H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
<ph_app_host>/api/projects/:project_id/ai_observability/traces/:id/review/

Example response

Status 200
RESPONSE
{
"review": {
"reviewedBy": "string",
"reviewedAt": "2019-08-24T14:15:22Z",
"comment": "string",
"scores": [
{
"name": "string",
"value": "string",
"label": "string"
}
]
},
"queue": {
"id": "string",
"name": "string",
"itemId": "string"
}
}
Status 400 The encoded trace id is malformed.