Self-driving with AI Observability
Contents
Self-driving is in open beta. It's improving quickly – expect rough edges, and expect them to disappear fast.
To power Self-driving, AI Observability turns your LLM data into signals: agents investigate what changed, and reports land in your inbox – where one click turns a finding into a pull request.
All of it needs LLM data flowing first – if you haven't instrumented yet, start with installation.
How it works
A signal is a structured finding: what's happening, with the evidence behind it. Your LLM data produces signals three ways:
| Signal | What it catches | When it runs |
|---|---|---|
| Eval reports | Quality and sentiment trends in production traffic | After each batch of evaluation results |
| Anomaly investigations (alpha) | Spikes in cost, latency, or errors | When an anomaly alert fires |
| Scouts | Regressions across cost, latency, errors, volume, and eval performance – or anything you describe | On a schedule you set |
Eval reports
Evaluations score your live generations in production – with an LLM judge, deterministic Hog code, or sentiment analysis. Eval reports are what make those scores actionable: instead of you watching a pass-rate chart, an agent reviews each batch of results and summarizes what it found, with example generations as evidence.
To set them up:
- Create an evaluation. PostHog provisions a report configuration automatically that you can further configure and have sent to email or Slack.
- Enable the AI observability source in your inbox settings, so report findings flow into your inbox as signals.
Anomaly investigations
Anomaly investigations are currently in alpha, as part of the product analytics signal source. We'd love to hear your feedback as we develop this feature.
Every LLM event behaves like a standard PostHog event, so cost, latency, and error rate are insights you can put an anomaly alert on. When one fires, an investigation agent picks it up before the notification reaches you: it digs into the underlying traces, writes up what it found, and suppresses the alert when the spike turns out to be a false positive – a batch job, a provider incident, one developer testing locally. You only read true positives.
To set them up:
- Create an insight on the metric you care about – cost, latency, or errors. The AI Observability dashboard charts are a good starting point.
- Add an anomaly detection alert to it – the detectors learn what normal looks like from your data. Fixed-threshold alerts fire like regular alerts, without an investigation.
- Enable the product analytics source in your inbox settings, so investigation findings land there as signals.
Scouts
Scouts are scheduled agents that explore your LLM data and file a report only when they find something noteworthy. They can be used to catch regressions across cost, latency, errors, volume, and eval performance, and remember your baselines between runs – so a persistent regression only shows up in a single report, not every day. AI Observability has a built-in scout ready to turn on.
Create one from the Self-driving tab in AI Observability or from your inbox: pick a template like the daily digest, or describe what's worth your attention in plain English.
What lands in your inbox
A report gives a summary and the evidence plus how actionable and important it is. If a fix is possible in code and the codebase is connected, one click opens a pull request. Reports are free; pull requests are priced per PR.
Next step
See the full picture of the loop your LLM data is feeding.