Evaluations

For instructions on how to authenticate to use this endpoint, see API overview.

Endpoints

GET
POST
GET
PATCH
DELETE
POST

List all evaluations

Required API key scopes

evaluation:read

Query parameters

  • directory_id
    string
  • directory_id__isnull
    boolean
  • enabled
    boolean
  • evaluation_type
    string
    One of: "hog""llm_judge""sentiment"
  • id__in
    array
  • limit
    integer
  • offset
    integer
  • order_by
    array
  • search
    string

Response


Example request

GET /api/projects/:project_id/evaluations
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/projects/:project_id/evaluations/

Example response

Status 200
RESPONSE
{
  "count": 123,
  "next": "http://api.example.org/accounts/?offset=400&limit=100",
  "previous": "http://api.example.org/accounts/?offset=200&limit=100",
  "results": [
    {
      "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
      "name": "string",
      "description": "string",
      "directory_id": "dca58562-24cf-4cbc-aa50-5a4039358aa5",
      "enabled": true,
      "status": "active",
      "status_reason": "provider_key_required",
      "status_reason_detail": "string",
      "evaluation_type": "llm_judge",
      "evaluation_config": {
        "prompt": "string"
      },
      "output_type": "boolean",
      "output_config": {
        "allows_na": false
      },
      "conditions": [
        {
          "id": "string",
          "rollout_percentage": 100,
          "properties": [
            {}
          ]
        }
      ],
      "target": "generation",
      "target_config": {
        "strategy": "fixed_window",
        "window_seconds": 10
      },
      "model_configuration": {
        "provider": "openai",
        "model": "string",
        "provider_key_id": "f265db88-9bcc-4e5b-add5-bfd9a815465c",
        "provider_key_name": "string"
      },
      "created_at": "2019-08-24T14:15:22Z",
      "updated_at": "2019-08-24T14:15:22Z",
      "created_by": {
        "id": 0,
        "uuid": "095be615-a8ad-4c33-8e9c-c7612fbf6c9f",
        "distinct_id": "string",
        "first_name": "string",
        "last_name": "string",
        "email": "user@example.com",
        "is_email_verified": true,
        "hedgehog_config": {},
        "role_at_organization": "engineering"
      },
      "deleted": true
    }
  ]
}

Create evaluations

Required API key scopes

evaluation:write

Request parameters

  • name
    string
  • description
    string
  • directory_id
    stringnull
  • enabled
    boolean
  • evaluation_type
  • evaluation_config
  • output_type
  • output_config
    object
  • conditions
    Click to open
    array
  • target
  • target_config
  • model_configuration
  • deleted
    boolean

Response


Example request

POST /api/projects/:project_id/evaluations
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl 
    -H 'Content-Type: application/json'\
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/projects/:project_id/evaluations/\
	-d name="string",\
	-d evaluation_type=undefined,\
	-d output_type=undefined

Example response

Status 201
RESPONSE
{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "name": "string",
  "description": "string",
  "directory_id": "dca58562-24cf-4cbc-aa50-5a4039358aa5",
  "enabled": true,
  "status": "active",
  "status_reason": "provider_key_required",
  "status_reason_detail": "string",
  "evaluation_type": "llm_judge",
  "evaluation_config": {
    "prompt": "string"
  },
  "output_type": "boolean",
  "output_config": {
    "allows_na": false
  },
  "conditions": [
    {
      "id": "string",
      "rollout_percentage": 100,
      "properties": [
        {}
      ]
    }
  ],
  "target": "generation",
  "target_config": {
    "strategy": "fixed_window",
    "window_seconds": 10
  },
  "model_configuration": {
    "provider": "openai",
    "model": "string",
    "provider_key_id": "f265db88-9bcc-4e5b-add5-bfd9a815465c",
    "provider_key_name": "string"
  },
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z",
  "created_by": {
    "id": 0,
    "uuid": "095be615-a8ad-4c33-8e9c-c7612fbf6c9f",
    "distinct_id": "string",
    "first_name": "string",
    "last_name": "string",
    "email": "user@example.com",
    "is_email_verified": true,
    "hedgehog_config": {},
    "role_at_organization": "engineering"
  },
  "deleted": true
}

Retrieve evaluations

Required API key scopes

evaluation:read

Path parameters

  • id
    string

Response


Example request

GET /api/projects/:project_id/evaluations/:id
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl \
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/projects/:project_id/evaluations/:id/

Example response

Status 200
RESPONSE
{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "name": "string",
  "description": "string",
  "directory_id": "dca58562-24cf-4cbc-aa50-5a4039358aa5",
  "enabled": true,
  "status": "active",
  "status_reason": "provider_key_required",
  "status_reason_detail": "string",
  "evaluation_type": "llm_judge",
  "evaluation_config": {
    "prompt": "string"
  },
  "output_type": "boolean",
  "output_config": {
    "allows_na": false
  },
  "conditions": [
    {
      "id": "string",
      "rollout_percentage": 100,
      "properties": [
        {}
      ]
    }
  ],
  "target": "generation",
  "target_config": {
    "strategy": "fixed_window",
    "window_seconds": 10
  },
  "model_configuration": {
    "provider": "openai",
    "model": "string",
    "provider_key_id": "f265db88-9bcc-4e5b-add5-bfd9a815465c",
    "provider_key_name": "string"
  },
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z",
  "created_by": {
    "id": 0,
    "uuid": "095be615-a8ad-4c33-8e9c-c7612fbf6c9f",
    "distinct_id": "string",
    "first_name": "string",
    "last_name": "string",
    "email": "user@example.com",
    "is_email_verified": true,
    "hedgehog_config": {},
    "role_at_organization": "engineering"
  },
  "deleted": true
}

Update evaluations

Required API key scopes

evaluation:write

Path parameters

  • id
    string

Request parameters

  • name
    string
  • description
    string
  • directory_id
    stringnull
  • enabled
    boolean
  • evaluation_type
  • evaluation_config
  • output_type
  • output_config
    object
  • conditions
    Click to open
    array
  • target
  • target_config
  • model_configuration
  • deleted
    boolean

Response


Example request

PATCH /api/projects/:project_id/evaluations/:id
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl -X PATCH \
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/projects/:project_id/evaluations/:id/\
	-d name="string"

Example response

Status 200
RESPONSE
{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "name": "string",
  "description": "string",
  "directory_id": "dca58562-24cf-4cbc-aa50-5a4039358aa5",
  "enabled": true,
  "status": "active",
  "status_reason": "provider_key_required",
  "status_reason_detail": "string",
  "evaluation_type": "llm_judge",
  "evaluation_config": {
    "prompt": "string"
  },
  "output_type": "boolean",
  "output_config": {
    "allows_na": false
  },
  "conditions": [
    {
      "id": "string",
      "rollout_percentage": 100,
      "properties": [
        {}
      ]
    }
  ],
  "target": "generation",
  "target_config": {
    "strategy": "fixed_window",
    "window_seconds": 10
  },
  "model_configuration": {
    "provider": "openai",
    "model": "string",
    "provider_key_id": "f265db88-9bcc-4e5b-add5-bfd9a815465c",
    "provider_key_name": "string"
  },
  "created_at": "2019-08-24T14:15:22Z",
  "updated_at": "2019-08-24T14:15:22Z",
  "created_by": {
    "id": 0,
    "uuid": "095be615-a8ad-4c33-8e9c-c7612fbf6c9f",
    "distinct_id": "string",
    "first_name": "string",
    "last_name": "string",
    "email": "user@example.com",
    "is_email_verified": true,
    "hedgehog_config": {},
    "role_at_organization": "engineering"
  },
  "deleted": true
}

Delete evaluations

Hard delete of this model is not allowed. Use a patch API call to set "deleted" to true

Required API key scopes

evaluation:write

Path parameters

  • id
    string

Example request

DELETE /api/projects/:project_id/evaluations/:id
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl  -X DELETE \
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/projects/:project_id/evaluations/:id/

Example response

Status 405 No response body

Create evaluations test hog

Test Hog evaluation code against sample events without saving.

Required API key scopes

evaluation:read

Request parameters

  • source
    string
  • sample_count
    integer
    Default: 5
  • allows_na
    boolean
    Default: false
  • conditions
    array
  • target
    Default: generation
  • target_config

Response


Example request

POST /api/projects/:project_id/evaluations/test_hog
export POSTHOG_PERSONAL_API_KEY=[your personal api key]
curl 
    -H 'Content-Type: application/json'\
    -H "Authorization: Bearer $POSTHOG_PERSONAL_API_KEY" \
    <ph_app_host>/api/projects/:project_id/evaluations/test_hog/\
	-d source="string"

Example response

Status 200
RESPONSE
{
  "results": [
    {
      "sample_id": "string",
      "sample_type": "generation",
      "event_uuid": "string",
      "trace_id": "string",
      "input_preview": "string",
      "output_preview": "string",
      "result": true,
      "reasoning": "string",
      "error": "string"
    }
  ],
  "message": "string"
}