Pocket guide to

AI Observability

Your AI features don't fail loudly. They fail plausibly and confidently, with nothing that looks like an error.

This pocket guide is about seeing what your model actually does in the wild, what it costs, and whether the outputs it returns are what users want.

Where to start

New to AI Observability? Start with AI Observability 101 on the next page – it explains what data is captured on every LLM call and how to interpret it.

Already instrumented? The use cases in Contents each include a prompt you can run in PostHog AI to build the thing the guide describes.

Either way, they all assume your LLM calls already reach PostHog. If they don't yet, run the wizard command, or install it yourself:

npx @posthog/wizard ai-observability

Contents

• AI Observability 10101
• Score your AI's answers with evals02
• Catch a code assistant inventing functions03
• Attribute AI cost to users and features04
• Measure whether users keep the answer05
• Debug a wrong answer with traces06