> AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt

# Tell an AI cost spike from real growth – Self-driving pocket guide

Canonical URL: https://posthog.com/docs/self-driving/pocket-guide/ai-spend-that-jumps

> **Cloud region:** This page uses US Cloud hosts. For an EU Cloud project, use `https://eu.posthog.com` instead of `https://app.posthog.com` (app host for links).

## On this page

- [What the scout looks at](#what-the-scout-looks-at)
- [The report in your inbox](#the-report-in-your-inbox)
- [The pull request you merge](#the-pull-request-you-merge)
- [The scout itself](#the-scout-itself)
- [The schedule it runs on](#the-schedule-it-runs-on)
- [Add this to your scout troop](#add-this-to-your-scout-troop)

Chapter 02

# Tell an AI cost spike from real growth – Self-driving pocket guide

An LLM bill moves for one of two reasons: more conversations, or more cost per conversation. The first is growth. The second is a regression – a prompt grew, a model changed, a cache stopped hitting, a retry looped, or someone abused your API for free inference.

This scout watches your LLM traces for regressions.

## What the scout looks at

You deployed a new version of your feature at `09:14`. Your conversation costs are rising, but volume is stable. Which case is it?

-   If both are rising together, it's just growth. The scout stays *quiet*.
-   If costs rise while volume is stable, it signals a possible regression. The scout ***reports***.

That second pattern is what the scout looks for – Fig. 1 shows the divergence it caught.

09:14 deploy

Conversations

flat at ~1,200/day

Cost per conversation

$0.09 → $0.28

Fig. 1 – What the scout saw: Your newest deploy went out at 09:14. Volume held, but cost per conversation didn't.

## The report in your inbox

When your product emits [signals](https://posthog.com/docs/self-driving/signals.md) that indicate AI costs are rising, a [scout](https://posthog.com/docs/self-driving/scouts.md) opens a [report](https://posthog.com/docs/self-driving/reports.md). Fig. 2 breaks the card down part by part. Click **Review** and the report opens Fig. 3 with the full evidence, the raw signals, and options to take action.

P11Cost per conversation up 3.1x while volume is flat 2Generations per conversation went 1.2 → 3.6 after the 09:14 deploy while tokens and models held steady3Scout · AI cost4ReadyActionable54h ago

ArchiveReview6

Fig. 2 – Once surfaced, your inbox surfaces this report card. The card includes its claim and triage, with the evidence sitting inside the report itself.Hover over the figure to learn about each element.

P1Cost per conversation up 3.1x while volume is flat 1ReadyActionableScout · AI cost· 4h ago

ArchiveCreate PR2

Summary3

Since yesterday's 09:14 deploy, conversations are flat at roughly 1,200/day but cost per conversation rose from $0.09 to $0.28. Generations per conversation went from 1.2 to 3.6 while input tokens and model distribution held steady. The extra generations carry non-zero cost and a `max_tokens` stop reason rather than a 429, so the provider billed for every retry – this is a retry loop, not a bigger prompt or a cache miss.

~$230/day above the 14-day norm

**Suggested fix:** Cap the retry count on the generation path and surface the underlying failure instead of retrying silently.

4

Evidence2 sources5

AI ObservabilityCost, tokens, model, cache reads, and error flags on every generation – enough to spot the change and attribute it.

Your codebaseDeploys, so a cost change can be tied to the commit that caused it rather than just a timestamp.

Fig. 3 – The report opened. From here, you can review the evidence and open a pull request.Hover over the figure to learn about each element.

## The pull request you merge

When you're ready to open a pull request, PostHog dispatches an agent to do it for you with the report and evidence attached as context. You review and merge it like any other PR – nothing merges without you.

Reports are free. Pull requests are **$15 each**, with your first three a month **free**. See self-driving [Pricing](https://posthog.com/docs/self-driving/pricing.md).

## The scout itself

Scouts are the heart of the self-driving system – they're instructions an agent follows on a schedule – Fig. 4 is the actual skill file for this guide's custom scout.

signals-scout-ai-cost/SKILL.md

```markdown
---
name: signals-scout-ai-cost
description: >
  Watches cost per conversation for AI features and reports when it breaks from its recent norm
  while volume holds steady, attributing the change to a model, prompt, cache, or retry cause.
allowed_tools:
  - emit_report
  - edit_report
---

# AI spend that jumps overnight

Watch unit economics for AI features, not the total bill.

Narrower than `signals-scout-ai-observability`, which rotates through cost, latency, errors,
volume and eval lenses and deliberately does not run every lens each tick. Run this one when
cost is the thing you want watched every day rather than whenever the rotation reaches it. If
you already run the canonical scout and cost regressions are surfacing in time, you don't need
both.

## Discriminator

Emit a report when cost per conversation rises above its trailing 14-day norm **while
conversation volume stays within its own normal range**. Write nothing when total cost rose but
cost per conversation is flat – that is usage growth.

### The bar

A starting threshold, to be tuned against what this team's data actually does:

- **≥1.5× the trailing 14-day median cost per conversation, sustained across two consecutive
  complete days**, or
- **≥2× for a single complete day**, where volume is within ±25% of its own 14-day median.

Use complete days only – a partial day reads as a spike every time. If the team's baseline is
genuinely noisy, widen the bar and write what you chose to the scratchpad rather than lowering
it and filing marginal reports.

## Explore

1. Compute daily cost per conversation over the last 14 days: total generation cost divided by
   distinct conversations – `$ai_trace_id`, or `$ai_session_id` where a conversation spans
   traces – split per feature.
2. Where a day clears the bar, decompose it. Check in this order, roughly the order of
   likelihood:
   - **Model changed** – compare the model distribution before and after.
   - **Cache hit rate dropped** – compare `$ai_cache_read_input_tokens` as a share of input
     tokens. A system-prompt edit invalidates the cache prefix and multiplies input cost with no
     change to model, token count, or generation count, so every other check here comes back
     clean while the bill climbs. Check it early.
   - **Prompt grew** – compare median input tokens per generation.
   - **Retries** – compare generations per conversation, and check `$ai_is_error` on the same
     generation stream.
   - **Output grew** – compare median output tokens.
3. If retries look like the cause, confirm the failures actually cost money. A 429 rejection
   bills nothing; only failures after the provider started generating do – timeouts mid-stream,
   `max_tokens` truncation, malformed tool output, guardrail rejections. Check `$ai_http_status`
   and `$ai_stop_reason`, and confirm the failed generations carry non-zero cost before calling
   retries the cause.
4. Name the most likely cause in the report. A cost report without a cause is just an invoice.
5. Where a deploy sits in the window, include the commit range.

## Disqualifiers

- Conversation volume moved as much as cost did – that's growth.
- The change is a deliberate, announced model upgrade.
- The change follows a deliberate caching-strategy change – a shortened prefix or a switched
  cache TTL raises unit cost on purpose.
- The provider changed its prices. This scout does not watch price lists, so confirm the
  per-token rate is unchanged before attributing a rise to your own code.
- The window covers a backfill, replay, or eval run rather than real traffic.
- The feature is new enough that there's no stable norm to compare against.
```

Show full example

Fig. 4 – The scout itself – the file an agent follows on a schedule.

It reads more than one [signal source](https://posthog.com/docs/self-driving/inbox/sources.md), so the report will name a likely cause, not just a number that moved.

This scout's signal sources come from the following product data:

-   [AI Observability](https://posthog.com/docs/ai-observability.md)

    Cost, tokens, model, cache reads, and error flags on every generation – enough to spot the change and attribute it.

-   [Your codebase](https://posthog.com/docs/self-driving/context.md)

    Deploys, so a cost change can be tied to the commit that caused it rather than just a timestamp.

## The schedule it runs on

This custom scout runs **daily** by default. You can choose to run it every 30 minutes up to once a day. It only writes when something is wrong, so most days it's silent.

## Add this to your scout troop

You'll need these to run this scout.

-   Required[AI Observability](https://posthog.com/docs/ai-observability/installation.md) instrumented, so $ai\_generation events are arriving
-   RequiredA property on generations identifying the feature, so you can split cost per surface
-   Recommended[Your codebase](https://posthog.com/docs/self-driving/context.md) connected via GitHub, to correlate a change against deploys

[Add this scout](https://app.posthog.com/inbox/config#createScout=eyJuYW1lIjoic2lnbmFscy1zY291dC1haS1jb3N0IiwiZGVzY3JpcHRpb24iOiJXYXRjaGVzIGNvc3QgcGVyIGNvbnZlcnNhdGlvbiBmb3IgQUkgZmVhdHVyZXMgYW5kIHJlcG9ydHMgd2hlbiBpdCBicmVha3MgZnJvbSBpdHMgcmVjZW50IG5vcm0gd2hpbGUgdm9sdW1lIGhvbGRzIHN0ZWFkeSwgYXR0cmlidXRpbmcgdGhlIGNoYW5nZSB0byBhIG1vZGVsLCBwcm9tcHQsIGNhY2hlLCBvciByZXRyeSBjYXVzZS5cbiIsImJvZHkiOiIjIEFJIHNwZW5kIHRoYXQganVtcHMgb3Zlcm5pZ2h0XG5cbldhdGNoIHVuaXQgZWNvbm9taWNzIGZvciBBSSBmZWF0dXJlcywgbm90IHRoZSB0b3RhbCBiaWxsLlxuXG5OYXJyb3dlciB0aGFuIGBzaWduYWxzLXNjb3V0LWFpLW9ic2VydmFiaWxpdHlgLCB3aGljaCByb3RhdGVzIHRocm91Z2ggY29zdCwgbGF0ZW5jeSwgZXJyb3JzLFxudm9sdW1lIGFuZCBldmFsIGxlbnNlcyBhbmQgZGVsaWJlcmF0ZWx5IGRvZXMgbm90IHJ1biBldmVyeSBsZW5zIGVhY2ggdGljay4gUnVuIHRoaXMgb25lIHdoZW5cbmNvc3QgaXMgdGhlIHRoaW5nIHlvdSB3YW50IHdhdGNoZWQgZXZlcnkgZGF5IHJhdGhlciB0aGFuIHdoZW5ldmVyIHRoZSByb3RhdGlvbiByZWFjaGVzIGl0LiBJZlxueW91IGFscmVhZHkgcnVuIHRoZSBjYW5vbmljYWwgc2NvdXQgYW5kIGNvc3QgcmVncmVzc2lvbnMgYXJlIHN1cmZhY2luZyBpbiB0aW1lLCB5b3UgZG9uJ3QgbmVlZFxuYm90aC5cblxuIyMgRGlzY3JpbWluYXRvclxuXG5FbWl0IGEgcmVwb3J0IHdoZW4gY29zdCBwZXIgY29udmVyc2F0aW9uIHJpc2VzIGFib3ZlIGl0cyB0cmFpbGluZyAxNC1kYXkgbm9ybSAqKndoaWxlXG5jb252ZXJzYXRpb24gdm9sdW1lIHN0YXlzIHdpdGhpbiBpdHMgb3duIG5vcm1hbCByYW5nZSoqLiBXcml0ZSBub3RoaW5nIHdoZW4gdG90YWwgY29zdCByb3NlIGJ1dFxuY29zdCBwZXIgY29udmVyc2F0aW9uIGlzIGZsYXQg4oCTIHRoYXQgaXMgdXNhZ2UgZ3Jvd3RoLlxuXG4jIyMgVGhlIGJhclxuXG5BIHN0YXJ0aW5nIHRocmVzaG9sZCwgdG8gYmUgdHVuZWQgYWdhaW5zdCB3aGF0IHRoaXMgdGVhbSdzIGRhdGEgYWN0dWFsbHkgZG9lczpcblxuLSAqKuKJpTEuNcOXIHRoZSB0cmFpbGluZyAxNC1kYXkgbWVkaWFuIGNvc3QgcGVyIGNvbnZlcnNhdGlvbiwgc3VzdGFpbmVkIGFjcm9zcyB0d28gY29uc2VjdXRpdmVcbiAgY29tcGxldGUgZGF5cyoqLCBvclxuLSAqKuKJpTLDlyBmb3IgYSBzaW5nbGUgY29tcGxldGUgZGF5KiosIHdoZXJlIHZvbHVtZSBpcyB3aXRoaW4gwrEyNSUgb2YgaXRzIG93biAxNC1kYXkgbWVkaWFuLlxuXG5Vc2UgY29tcGxldGUgZGF5cyBvbmx5IOKAkyBhIHBhcnRpYWwgZGF5IHJlYWRzIGFzIGEgc3Bpa2UgZXZlcnkgdGltZS4gSWYgdGhlIHRlYW0ncyBiYXNlbGluZSBpc1xuZ2VudWluZWx5IG5vaXN5LCB3aWRlbiB0aGUgYmFyIGFuZCB3cml0ZSB3aGF0IHlvdSBjaG9zZSB0byB0aGUgc2NyYXRjaHBhZCByYXRoZXIgdGhhbiBsb3dlcmluZ1xuaXQgYW5kIGZpbGluZyBtYXJnaW5hbCByZXBvcnRzLlxuXG4jIyBFeHBsb3JlXG5cbjEuIENvbXB1dGUgZGFpbHkgY29zdCBwZXIgY29udmVyc2F0aW9uIG92ZXIgdGhlIGxhc3QgMTQgZGF5czogdG90YWwgZ2VuZXJhdGlvbiBjb3N0IGRpdmlkZWQgYnlcbiAgIGRpc3RpbmN0IGNvbnZlcnNhdGlvbnMg4oCTIGAkYWlfdHJhY2VfaWRgLCBvciBgJGFpX3Nlc3Npb25faWRgIHdoZXJlIGEgY29udmVyc2F0aW9uIHNwYW5zXG4gICB0cmFjZXMg4oCTIHNwbGl0IHBlciBmZWF0dXJlLlxuMi4gV2hlcmUgYSBkYXkgY2xlYXJzIHRoZSBiYXIsIGRlY29tcG9zZSBpdC4gQ2hlY2sgaW4gdGhpcyBvcmRlciwgcm91Z2hseSB0aGUgb3JkZXIgb2ZcbiAgIGxpa2VsaWhvb2Q6XG4gICAtICoqTW9kZWwgY2hhbmdlZCoqIOKAkyBjb21wYXJlIHRoZSBtb2RlbCBkaXN0cmlidXRpb24gYmVmb3JlIGFuZCBhZnRlci5cbiAgIC0gKipDYWNoZSBoaXQgcmF0ZSBkcm9wcGVkKiog4oCTIGNvbXBhcmUgYCRhaV9jYWNoZV9yZWFkX2lucHV0X3Rva2Vuc2AgYXMgYSBzaGFyZSBvZiBpbnB1dFxuICAgICB0b2tlbnMuIEEgc3lzdGVtLXByb21wdCBlZGl0IGludmFsaWRhdGVzIHRoZSBjYWNoZSBwcmVmaXggYW5kIG11bHRpcGxpZXMgaW5wdXQgY29zdCB3aXRoIG5vXG4gICAgIGNoYW5nZSB0byBtb2RlbCwgdG9rZW4gY291bnQsIG9yIGdlbmVyYXRpb24gY291bnQsIHNvIGV2ZXJ5IG90aGVyIGNoZWNrIGhlcmUgY29tZXMgYmFja1xuICAgICBjbGVhbiB3aGlsZSB0aGUgYmlsbCBjbGltYnMuIENoZWNrIGl0IGVhcmx5LlxuICAgLSAqKlByb21wdCBncmV3Kiog4oCTIGNvbXBhcmUgbWVkaWFuIGlucHV0IHRva2VucyBwZXIgZ2VuZXJhdGlvbi5cbiAgIC0gKipSZXRyaWVzKiog4oCTIGNvbXBhcmUgZ2VuZXJhdGlvbnMgcGVyIGNvbnZlcnNhdGlvbiwgYW5kIGNoZWNrIGAkYWlfaXNfZXJyb3JgIG9uIHRoZSBzYW1lXG4gICAgIGdlbmVyYXRpb24gc3RyZWFtLlxuICAgLSAqKk91dHB1dCBncmV3Kiog4oCTIGNvbXBhcmUgbWVkaWFuIG91dHB1dCB0b2tlbnMuXG4zLiBJZiByZXRyaWVzIGxvb2sgbGlrZSB0aGUgY2F1c2UsIGNvbmZpcm0gdGhlIGZhaWx1cmVzIGFjdHVhbGx5IGNvc3QgbW9uZXkuIEEgNDI5IHJlamVjdGlvblxuICAgYmlsbHMgbm90aGluZzsgb25seSBmYWlsdXJlcyBhZnRlciB0aGUgcHJvdmlkZXIgc3RhcnRlZCBnZW5lcmF0aW5nIGRvIOKAkyB0aW1lb3V0cyBtaWQtc3RyZWFtLFxuICAgYG1heF90b2tlbnNgIHRydW5jYXRpb24sIG1hbGZvcm1lZCB0b29sIG91dHB1dCwgZ3VhcmRyYWlsIHJlamVjdGlvbnMuIENoZWNrIGAkYWlfaHR0cF9zdGF0dXNgXG4gICBhbmQgYCRhaV9zdG9wX3JlYXNvbmAsIGFuZCBjb25maXJtIHRoZSBmYWlsZWQgZ2VuZXJhdGlvbnMgY2Fycnkgbm9uLXplcm8gY29zdCBiZWZvcmUgY2FsbGluZ1xuICAgcmV0cmllcyB0aGUgY2F1c2UuXG40LiBOYW1lIHRoZSBtb3N0IGxpa2VseSBjYXVzZSBpbiB0aGUgcmVwb3J0LiBBIGNvc3QgcmVwb3J0IHdpdGhvdXQgYSBjYXVzZSBpcyBqdXN0IGFuIGludm9pY2UuXG41LiBXaGVyZSBhIGRlcGxveSBzaXRzIGluIHRoZSB3aW5kb3csIGluY2x1ZGUgdGhlIGNvbW1pdCByYW5nZS5cblxuIyMgRGlzcXVhbGlmaWVyc1xuXG4tIENvbnZlcnNhdGlvbiB2b2x1bWUgbW92ZWQgYXMgbXVjaCBhcyBjb3N0IGRpZCDigJMgdGhhdCdzIGdyb3d0aC5cbi0gVGhlIGNoYW5nZSBpcyBhIGRlbGliZXJhdGUsIGFubm91bmNlZCBtb2RlbCB1cGdyYWRlLlxuLSBUaGUgY2hhbmdlIGZvbGxvd3MgYSBkZWxpYmVyYXRlIGNhY2hpbmctc3RyYXRlZ3kgY2hhbmdlIOKAkyBhIHNob3J0ZW5lZCBwcmVmaXggb3IgYSBzd2l0Y2hlZFxuICBjYWNoZSBUVEwgcmFpc2VzIHVuaXQgY29zdCBvbiBwdXJwb3NlLlxuLSBUaGUgcHJvdmlkZXIgY2hhbmdlZCBpdHMgcHJpY2VzLiBUaGlzIHNjb3V0IGRvZXMgbm90IHdhdGNoIHByaWNlIGxpc3RzLCBzbyBjb25maXJtIHRoZVxuICBwZXItdG9rZW4gcmF0ZSBpcyB1bmNoYW5nZWQgYmVmb3JlIGF0dHJpYnV0aW5nIGEgcmlzZSB0byB5b3VyIG93biBjb2RlLlxuLSBUaGUgd2luZG93IGNvdmVycyBhIGJhY2tmaWxsLCByZXBsYXksIG9yIGV2YWwgcnVuIHJhdGhlciB0aGFuIHJlYWwgdHJhZmZpYy5cbi0gVGhlIGZlYXR1cmUgaXMgbmV3IGVub3VnaCB0aGF0IHRoZXJlJ3Mgbm8gc3RhYmxlIG5vcm0gdG8gY29tcGFyZSBhZ2FpbnN0LiIsInNvdXJjZSI6InBvY2tldF9ndWlkZSJ9)Opens PostHog with it prefilled. Review it, then hit Create.

Not set up yet? One command installs PostHog, connects GitHub, and turns on a default troop of scouts. That troop doesn't include this one – add it above once you're set up.

```
npx @posthog/wizard self-driving
```