Llm

For instructions on how to authenticate to use this endpoint, see API overview.

Endpoints

GET
POST
GET
PATCH
DELETE
GET
POST
GET
PATCH
DELETE
GET
POST
POST
GET
POST
GET
POST
GET

List all environments dataset items

Required API key scopes

dataset:read

Path parameters

  • environment_id
    string

Query parameters

  • dataset
    string
  • limit
    integer
  • offset
    integer

Response


Example request

GET /api/environments/:environment_id/dataset_items
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:environment_id/dataset_items/

Example response

Status 200
RESPONSE
{
  "count": 123,
  "next": "http://api.example.org/accounts/?offset=400&limit=100",
  "previous": "http://api.example.org/accounts/?offset=200&limit=100",
  "results": [
    {
      "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
      "dataset": "d93f2be0-02b1-4d87-8b6e-fc5cba21ed8d",
      "input": null,
      "output": null,
      "metadata": null,
      "ref_trace_id": "string",
      "ref_timestamp": "2019-08-24T14:15:22Z",
      "ref_source_id": "string",
      "deleted": true,
      "created_at": "2019-08-24T14:15:22Z",
      "updated_at": "2019-08-24T14:15:22Z",
      "created_by": {
        "id": 0,
        "uuid": "095be615-a8ad-4c33-8e9c-c7612fbf6c9f",
        "distinct_id": "string",
        "first_name": "string",
        "last_name": "string",
        "email": "user@example.com",
        "is_email_verified": true,
        "hedgehog_config": {
          "property1": null,
          "property2": null
        },
        "role_at_organization": "engineering"
      },
      "team": 0
    }
  ]
}

Create environments dataset items

Required API key scopes

dataset:write

Path parameters

  • environment_id
    string

Request parameters

  • dataset
    string
  • input
  • output
  • metadata
  • ref_trace_id
    string
  • ref_timestamp
    string
  • ref_source_id
    string
  • deleted
    boolean

Response


Example request

POST /api/environments/:environment_id/dataset_items
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl 
    -H 'Content-Type: application/json'\
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:environment_id/dataset_items/\
	-d dataset="string"

Example response

Status 201
RESPONSE
{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "dataset": "d93f2be0-02b1-4d87-8b6e-fc5cba21ed8d",
  "input": null,
  "output": null,
  "metadata": null,
  "ref_trace_id": "string",
  "ref_timestamp": "2019-08-24T14:15:22Z",
  "ref_source_id": "string",
  "deleted": true,
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z",
  "created_by": {
    "id": 0,
    "uuid": "095be615-a8ad-4c33-8e9c-c7612fbf6c9f",
    "distinct_id": "string",
    "first_name": "string",
    "last_name": "string",
    "email": "user@example.com",
    "is_email_verified": true,
    "hedgehog_config": {
      "property1": null,
      "property2": null
    },
    "role_at_organization": "engineering"
  },
  "team": 0
}

Retrieve environments dataset items

Required API key scopes

dataset:read

Path parameters

  • environment_id
    string
  • id
    string

Response


Example request

GET /api/environments/:environment_id/dataset_items/:id
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:environment_id/dataset_items/:id/

Example response

Status 200
RESPONSE
{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "dataset": "d93f2be0-02b1-4d87-8b6e-fc5cba21ed8d",
  "input": null,
  "output": null,
  "metadata": null,
  "ref_trace_id": "string",
  "ref_timestamp": "2019-08-24T14:15:22Z",
  "ref_source_id": "string",
  "deleted": true,
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z",
  "created_by": {
    "id": 0,
    "uuid": "095be615-a8ad-4c33-8e9c-c7612fbf6c9f",
    "distinct_id": "string",
    "first_name": "string",
    "last_name": "string",
    "email": "user@example.com",
    "is_email_verified": true,
    "hedgehog_config": {
      "property1": null,
      "property2": null
    },
    "role_at_organization": "engineering"
  },
  "team": 0
}

Update environments dataset items

Required API key scopes

dataset:write

Path parameters

  • environment_id
    string
  • id
    string

Request parameters

  • dataset
    string
  • input
  • output
  • metadata
  • ref_trace_id
    string
  • ref_timestamp
    string
  • ref_source_id
    string
  • deleted
    boolean

Response


Example request

PATCH /api/environments/:environment_id/dataset_items/:id
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl -X PATCH \
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:environment_id/dataset_items/:id/\
	-d dataset="string"

Example response

Status 200
RESPONSE
{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "dataset": "d93f2be0-02b1-4d87-8b6e-fc5cba21ed8d",
  "input": null,
  "output": null,
  "metadata": null,
  "ref_trace_id": "string",
  "ref_timestamp": "2019-08-24T14:15:22Z",
  "ref_source_id": "string",
  "deleted": true,
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z",
  "created_by": {
    "id": 0,
    "uuid": "095be615-a8ad-4c33-8e9c-c7612fbf6c9f",
    "distinct_id": "string",
    "first_name": "string",
    "last_name": "string",
    "email": "user@example.com",
    "is_email_verified": true,
    "hedgehog_config": {
      "property1": null,
      "property2": null
    },
    "role_at_organization": "engineering"
  },
  "team": 0
}

Delete environments dataset items

Hard delete of this model is not allowed. Use a patch API call to set "deleted" to true

Required API key scopes

dataset:write

Path parameters

  • environment_id
    string
  • id
    string

Example request

DELETE /api/environments/:environment_id/dataset_items/:id
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl  -X DELETE \
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:environment_id/dataset_items/:id/

Example response

Status 405 No response body

List all environments datasets

Required API key scopes

dataset:read

Path parameters

  • environment_id
    string

Query parameters

  • id__in
    array
  • limit
    integer
  • offset
    integer
  • order_by
    array
  • search
    string

Response


Example request

GET /api/environments/:environment_id/datasets
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:environment_id/datasets/

Example response

Status 200
RESPONSE
{
  "count": 123,
  "next": "http://api.example.org/accounts/?offset=400&limit=100",
  "previous": "http://api.example.org/accounts/?offset=200&limit=100",
  "results": [
    {
      "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
      "name": "string",
      "description": "string",
      "metadata": null,
      "created_at": "2019-08-24T14:15:22Z",
      "updated_at": "2019-08-24T14:15:22Z",
      "deleted": true,
      "created_by": {
        "id": 0,
        "uuid": "095be615-a8ad-4c33-8e9c-c7612fbf6c9f",
        "distinct_id": "string",
        "first_name": "string",
        "last_name": "string",
        "email": "user@example.com",
        "is_email_verified": true,
        "hedgehog_config": {
          "property1": null,
          "property2": null
        },
        "role_at_organization": "engineering"
      },
      "team": 0
    }
  ]
}

Create environments datasets

Required API key scopes

dataset:write

Path parameters

  • environment_id
    string

Request parameters

  • name
    string
  • description
    string
  • metadata
  • deleted
    boolean

Response


Example request

POST /api/environments/:environment_id/datasets
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl 
    -H 'Content-Type: application/json'\
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:environment_id/datasets/\
	-d name="string"

Example response

Status 201
RESPONSE
{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "name": "string",
  "description": "string",
  "metadata": null,
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z",
  "deleted": true,
  "created_by": {
    "id": 0,
    "uuid": "095be615-a8ad-4c33-8e9c-c7612fbf6c9f",
    "distinct_id": "string",
    "first_name": "string",
    "last_name": "string",
    "email": "user@example.com",
    "is_email_verified": true,
    "hedgehog_config": {
      "property1": null,
      "property2": null
    },
    "role_at_organization": "engineering"
  },
  "team": 0
}

Retrieve environments datasets

Required API key scopes

dataset:read

Path parameters

  • environment_id
    string
  • id
    string

Response


Example request

GET /api/environments/:environment_id/datasets/:id
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:environment_id/datasets/:id/

Example response

Status 200
RESPONSE
{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "name": "string",
  "description": "string",
  "metadata": null,
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z",
  "deleted": true,
  "created_by": {
    "id": 0,
    "uuid": "095be615-a8ad-4c33-8e9c-c7612fbf6c9f",
    "distinct_id": "string",
    "first_name": "string",
    "last_name": "string",
    "email": "user@example.com",
    "is_email_verified": true,
    "hedgehog_config": {
      "property1": null,
      "property2": null
    },
    "role_at_organization": "engineering"
  },
  "team": 0
}

Update environments datasets

Required API key scopes

dataset:write

Path parameters

  • environment_id
    string
  • id
    string

Request parameters

  • name
    string
  • description
    string
  • metadata
  • deleted
    boolean

Response


Example request

PATCH /api/environments/:environment_id/datasets/:id
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl -X PATCH \
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:environment_id/datasets/:id/\
	-d name="string"

Example response

Status 200
RESPONSE
{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "name": "string",
  "description": "string",
  "metadata": null,
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z",
  "deleted": true,
  "created_by": {
    "id": 0,
    "uuid": "095be615-a8ad-4c33-8e9c-c7612fbf6c9f",
    "distinct_id": "string",
    "first_name": "string",
    "last_name": "string",
    "email": "user@example.com",
    "is_email_verified": true,
    "hedgehog_config": {
      "property1": null,
      "property2": null
    },
    "role_at_organization": "engineering"
  },
  "team": 0
}

Delete environments datasets

Hard delete of this model is not allowed. Use a patch API call to set "deleted" to true

Required API key scopes

dataset:write

Path parameters

  • environment_id
    string
  • id
    string

Example request

DELETE /api/environments/:environment_id/datasets/:id
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl  -X DELETE \
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:environment_id/datasets/:id/

Example response

Status 405 No response body

Retrieve llm analytics evaluation config

Get the evaluation config for this team

Required API key scopes

llm_provider_key:read

Path parameters

  • project_id
    string

Example request

GET /api/environments/:project_id/llm_analytics/evaluation_config
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:project_id/llm_analytics/evaluation_config/

Example response

Status 200 No response body

Create llm analytics evaluation config set active key

Set the active provider key for evaluations

Path parameters

  • project_id
    string

Example request

POST /api/environments/:project_id/llm_analytics/evaluation_config/set_active_key
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl 
    -H 'Content-Type: application/json'\
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:project_id/llm_analytics/evaluation_config/set_active_key/

Example response

Status 200 No response body

Create llm analytics evaluation summary

Generate an AI-powered summary of evaluation results.

This endpoint analyzes evaluation runs and identifies patterns in passing and failing evaluations, providing actionable recommendations.

Data is fetched server-side by evaluation ID to ensure data integrity.

Use Cases:

  • Understand why evaluations are passing or failing
  • Identify systematic issues in LLM responses
  • Get recommendations for improving response quality
  • Review patterns across many evaluation runs at once

Required API key scopes

llm_analytics:write

Path parameters

  • project_id
    string

Request parameters

  • evaluation_id
    string
  • filter
    Default: all
  • generation_ids
    array
  • force_refresh
    boolean
    Default: false

Response


Example request

POST /api/environments/:project_id/llm_analytics/evaluation_summary
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl 
    -H 'Content-Type: application/json'\
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:project_id/llm_analytics/evaluation_summary/\
	-d evaluation_id="string"

Example response

Status 200
RESPONSE
{
  "overall_assessment": "Evaluations show generally good quality with some factual accuracy issues.",
  "pass_patterns": [
    {
      "title": "Clear Communication",
      "description": "Responses consistently provided well-structured information",
      "frequency": "common",
      "example_generation_ids": [
        "gen_abc123",
        "gen_ghi789"
      ]
    }
  ],
  "fail_patterns": [
    {
      "title": "Factual Errors",
      "description": "Some responses contained inaccurate information",
      "frequency": "occasional",
      "example_generation_ids": [
        "gen_def456"
      ]
    }
  ],
  "na_patterns": [],
  "recommendations": [
    "Implement fact-checking for critical claims",
    "Add source citations where applicable"
  ],
  "statistics": {
    "total_analyzed": 3,
    "pass_count": 2,
    "fail_count": 1,
    "na_count": 0
  }
}
Status 400
Status 403
Status 404
Status 500

Retrieve llm analytics models

List available models for a provider.

Required API key scopes

llm_provider_key:read

Path parameters

  • project_id
    string

Example request

GET /api/environments/:project_id/llm_analytics/models
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:project_id/llm_analytics/models/

Example response

Status 200 No response body

Create llm analytics provider key validations

Validate LLM provider API keys without persisting them

Required API key scopes

llm_provider_key:write

Path parameters

  • project_id
    string

Example request

POST /api/environments/:project_id/llm_analytics/provider_key_validations
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl 
    -H 'Content-Type: application/json'\
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:project_id/llm_analytics/provider_key_validations/

Example response

Status 201 No response body

List all llm analytics provider keys

Required API key scopes

llm_provider_key:read

Path parameters

  • project_id
    string

Query parameters

  • limit
    integer
  • offset
    integer

Response


Example request

GET /api/environments/:project_id/llm_analytics/provider_keys
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:project_id/llm_analytics/provider_keys/

Example response

Status 200
RESPONSE
{
  "count": 123,
  "next": "http://api.example.org/accounts/?offset=400&limit=100",
  "previous": "http://api.example.org/accounts/?offset=200&limit=100",
  "results": [
    {
      "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
      "provider": "openai",
      "name": "string",
      "state": "unknown",
      "error_message": "string",
      "api_key": "string",
      "api_key_masked": "string",
      "set_as_active": false,
      "created_at": "2019-08-24T14:15:22Z",
      "created_by": {
        "id": 0,
        "uuid": "095be615-a8ad-4c33-8e9c-c7612fbf6c9f",
        "distinct_id": "string",
        "first_name": "string",
        "last_name": "string",
        "email": "user@example.com",
        "is_email_verified": true,
        "hedgehog_config": {
          "property1": null,
          "property2": null
        },
        "role_at_organization": "engineering"
      },
      "last_used_at": "2019-08-24T14:15:22Z"
    }
  ]
}

Create llm analytics provider keys

Required API key scopes

llm_provider_key:write

Path parameters

  • project_id
    string

Request parameters

  • provider
  • name
    string
  • api_key
    string
  • set_as_active
    boolean
    Default: false

Response


Example request

POST /api/environments/:project_id/llm_analytics/provider_keys
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl 
    -H 'Content-Type: application/json'\
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:project_id/llm_analytics/provider_keys/\
	-d provider=undefined,\
	-d name="string"

Example response

Status 201
RESPONSE
{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "provider": "openai",
  "name": "string",
  "state": "unknown",
  "error_message": "string",
  "api_key": "string",
  "api_key_masked": "string",
  "set_as_active": false,
  "created_at": "2019-08-24T14:15:22Z",
  "created_by": {
    "id": 0,
    "uuid": "095be615-a8ad-4c33-8e9c-c7612fbf6c9f",
    "distinct_id": "string",
    "first_name": "string",
    "last_name": "string",
    "email": "user@example.com",
    "is_email_verified": true,
    "hedgehog_config": {
      "property1": null,
      "property2": null
    },
    "role_at_organization": "engineering"
  },
  "last_used_at": "2019-08-24T14:15:22Z"
}

Retrieve llm analytics provider keys

Required API key scopes

llm_provider_key:read

Path parameters

  • id
    string
  • project_id
    string

Response


Example request

GET /api/environments/:project_id/llm_analytics/provider_keys/:id
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/environments/:project_id/llm_analytics/provider_keys/:id/

Example response

Status 200
RESPONSE
{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "provider": "openai",
  "name": "string",
  "state": "unknown",
  "error_message": "string",
  "api_key": "string",
  "api_key_masked": "string",
  "set_as_active": false,
  "created_at": "2019-08-24T14:15:22Z",
  "created_by": {
    "id": 0,
    "uuid": "095be615-a8ad-4c33-8e9c-c7612fbf6c9f",
    "distinct_id": "string",
    "first_name": "string",
    "last_name": "string",
    "email": "user@example.com",
    "is_email_verified": true,
    "hedgehog_config": {
      "property1": null,
      "property2": null
    },
    "role_at_organization": "engineering"
  },
  "last_used_at": "2019-08-24T14:15:22Z"
}
Next page →

Community questions

Questions about this page? or post a community question.