Agent skills for AI Observability

Contents

PostHog ships agent skills for AI Observability that teach your coding agent how to investigate LLM traces, costs, clusters, evaluations, and failures through the PostHog MCP server. They're the same skills PostHog's own background agents run.

Set up

  1. Make sure there's LLM data to investigate – if you haven't instrumented yet, install a PostHog SDK or provider integration.

  2. Connect the PostHog MCP server. (The AI plugin in step 3 does this for you.)

  3. Install the PostHog AI plugin – it bundles every PostHog skill and keeps them up to date. Its README covers installation in Claude Code, Codex, Cursor, and Gemini CLI.

Want the skills directly?

Every PostHog skill also ships as skills.zip on the latest skills release – ready-to-use SKILL.md directories you can drop into your agent's skills folder (like .claude/skills/ for Claude Code).

Each skill's description tells your agent when to load it, so "why did our LLM costs spike yesterday?" picks up exploring-llm-costs without you naming it.

The skills

analyzing-expensive-users

View source

Analyze the most expensive users in AI observability and explain why they cost so much.

  • execute-sql
  • query-llm-traces-list
  • query-llm-trace
  • read-data-schema
  • generate-app-url

creating-online-evaluations

View source

Author continuously-running online evaluations in PostHog AI observability, grounded in real failure modes you've identified.

  • llma-evaluation-config-get
  • llma-provider-key-list
  • llma-evaluation-judge-models
  • llma-evaluation-directory-list
  • llma-evaluation-directory-create
  • llma-evaluation-test-hog
  • llma-evaluation-create
  • llma-evaluation-run
  • llma-evaluation-update
  • execute-sql
  • generate-app-url

exploring-ai-failures

View source

Find where an AI/LLM application is failing in production and surface the failure patterns, working from real traces.

  • query-llm-traces-list
  • query-llm-trace
  • execute-sql
  • llma-evaluation-list
  • llma-evaluation-summary-create
  • generate-app-url

exploring-llm-clusters

View source

Investigate AI observability clusters — understand usage patterns in AI/LLM traffic, compare cluster behavior, compute cost/latency metrics, and drill into individual traces within clusters.

  • llma-clustering-job-list
  • llma-clustering-job-get
  • execute-sql
  • query-llm-traces-list
  • query-llm-trace

exploring-llm-costs

View source

Investigate LLM spend in PostHog — total cost over time, cost by model, provider, user, trace, or custom dimension, token and cache-hit economics, and cost regressions.

  • execute-sql
  • query-llm-traces-list
  • query-llm-trace
  • read-data-schema
  • insight-create
  • dashboard-create
  • alert-create
  • docs-search

exploring-llm-evaluations

View source

Investigate AI observability evaluations — hog (deterministic code-based), llm_judge (LLM-prompt-based), and sentiment (user-message sentiment).

  • llma-evaluation-list
  • llma-evaluation-get
  • llma-evaluation-create
  • llma-evaluation-update
  • llma-evaluation-delete
  • llma-evaluation-run
  • llma-evaluation-test-hog
  • llma-evaluation-summary-create
  • execute-sql
  • query-llm-trace

exploring-llm-traces

View source

ABSOLUTE MUST to debug and inspect LLM/AI agent traces using PostHog's MCP tools.

  • query-llm-traces-list
  • query-llm-trace
  • read-data-schema
  • execute-sql

Prefer PostHog to run these investigations for you and offer fixes when possible? Checkout self-driving.

Was this page useful?