> AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt

# Find costly AI users – Self-driving pocket guide

[](/pocket-guides.md)Aa

[](/pocket-guides/self-driving/silent-failure-core-action.md)[](/pocket-guides/self-driving/pages-people-cant-leave.md)

# Find costly AI users – Self-driving pocket guide

Tokenmaxxing is so in right now. But tokenmaxxing and value-minning? Less cool.

A token leaderboard shows you your heaviest users – it doesn't show you your most expensive ones. A costly user is the one that should worry you: same work, same impact, more money.

## What makes a user expensive

An expensive user isn't one who uses your product often – it's one who costs more without generating more usage or value. What triggers the cost increase is how you tell the difference.

**Something you deployed triggered the cost change.** A model swap or a broken cache prefix triggers this. This is something you can spot on your AI bill because everyone's cost moves on the same day. [Tell an AI cost spike from real growth](/pocket-guides/self-driving/ai-spend-that-jumps.md) is the chapter that covers this.

**The user's input caused it.** A 40-page PDF or six-hour conversation. Nothing changed in your code, but what it costs to serve that user did. This is the tokenmaxxing, value-minning user – and where you're actually losing money.

## Divide cost by use

Your top spender is at $400 this month. Is that good or bad?

You probably have no idea because spend is two numbers multiplied together: how much someone used your product, and what each use cost you.

To figure it out, you should divide cost by use. PostHog already has the data you need to do this calculation – every AI observability event carries `$ai_trace_id`, and one [trace](/docs/ai-observability/traces.md) is one full interaction:

-   The model call
-   Retrieval
-   Tool calls

So you already have a denominator to divide by, and that $400 splits into two very different stories. In Fig. 1, two users have the same spend, but User A is the one using your AI product efficiently.

Monthly spend = Traces×Cost per trace

| Monthly spend | Traces | Cost per trace |
| --- | --- | --- |
| User A | $400 | 10,000 | $0.04 | Growth |
| User B | $400 | 3,000 | $0.13 | Regression |

Fig. 1 – Two users, the same monthly spend, and only one of them is a problem. The bar is cost per trace.

This approach works well for traces. If you're building a multi-turn experience like a chatbot, use `$ai_session_id` instead. It groups traces into a whole conversation, and cost per session is the number that matters.

## Averages hide expensive users

Watching cost per trace across your whole product works well for regressions that hit everyone, like a bad deploy or a model swap.

This doesn't reveal edge cases and outliers. A retry loop that only fires on documents over 8k tokens hits the handful of accounts that send long documents. Average that in with thousands of healthy traces and it disappears.

So one comparison isn't enough. You need two: how a user's cost per trace compares to everyone else's, and how it compares to their own last week. A user who doubled against themselves is worth a look even if they're nowhere near the top of your bill.

## Anatomy of a trace

Back to User B in Fig. 1: cost per trace up 3x, routed through the same workflow. You know where the spend is going. You just don't know why yet.

So you open User B's trace, and Fig. 2 is what you find.

$ai\_tracedocument\_qa200 OK · 14.2s · $0.131$ai\_spanload\_document9,200 tokens in$ai\_generationextract\_fieldsstop: max\_tokens · $0.0241$ai\_generationextract\_fields · retry 1stop: max\_tokens · $0.024$ai\_generationextract\_fields · retry 2stop: max\_tokens · $0.025$ai\_generationextract\_fields · retry 3stop: max\_tokens · $0.024$ai\_generationanswer\_questionstop: end\_turn · $0.0342

Fig. 2 – One request from User B. Four attempts at the extraction step, every one of them billed, and one answer at the end.

There's 5 [generations](/docs/ai-observability/generations.md) in a request that should take one, against a population median of 1.3. *There's your retry loop.*

Except a retry loop isn't automatically expensive. A 429 rejection bills nothing, because the provider never started generating. A generation that ran, produced tokens, and hit `max_tokens` bills in full. So check the stop reason before you blame the retries.

Every attempt here stopped on `max_tokens`, and every one carries cost. That's the finding: the provider billed four times to produce nothing, because the extraction step retries on malformed output and never gives up loudly.

Now you know what to fix.

## You are not going to do this every morning

No one has time to rank users, work out the population baseline, compare every candidate against it, split by model and workflow, and examine their traces? And you need to do this every day.

So hand it to a [scout](/docs/self-driving/scouts.md). A scout runs on a schedule and you only hear about it when something needs your attention.

When it finds something, you get a [report](/docs/self-driving/reports.md) with the comparison window, the numbers you need to judge it, the cause it found in the traces, and what action to take. Fig. 3

Inbox

1 unread

1

Scout · costly users·3h ago

### One customer's cost per trace tripled after switching to the document workflow

Four accounts moved from $0.04 to $0.13 cost per trace over two days while their trace volume held steady. All four route through the `document-qa` workflow, which retries the extraction step on malformed output without a cap.

Traces show a median of 5 generations per request against a population median of 1.3, and the retried generations carry non-zero cost with a `max_tokens` stop reason. The provider billed every retry.

4 accounts, ~$180/day above their own baseline

**Suggested action:**

Cap the retry count on the `document-qa` extraction step and fail loudly instead of retrying on malformed output.

An agent opens the pull request. You review and merge.

2

3

Runs daily. Every earlier run this week wrote nothing.

Fig. 3 – What lands in your inbox when the scout finds it.Hover over the figure to learn about each element.

## Add the scout

The scout itself is just a file an agent follows on a schedule, and Fig. 4 is the one this guide is about.

signals-scout-ai-observability-costly-users/SKILL.md

```markdown
# AI observability costly or unusual users
Find extraordinary user-level cost or usage patterns in the most recent complete 24 hours. Compare them with the preceding 24 hours, the recent 7-day baseline, and the same weekday when traffic is seasonal. Never compare a complete period with a partial one.
A user ranking highly by spend is not a finding. Look for a material departure in unit economics or behavior with a controllable cause, validated in real traces. High volume with normal cost per trace may be healthy usage.
## Use the packaged analysis skills
Load these preinstalled skills through the runtime's packaged-skill mechanism when relevant:
- `analyzing-expensive-users`
- `exploring-llm-costs`
- `exploring-llm-traces`
- `querying-posthog-data`
These are packaged runtime skills, not project skill-store entries. Do not use `skill-list` or `skill-get` to load them.
## Avoid duplicate work
Read this Scout's last 14 days of run summaries with `scout-runs-list`, filtered by its exact `skill_name` and current `skill_version`. Retrieve relevant details with `scout-runs-retrieve`.
Search the scratchpad and recent Inbox reports for the user, segment, workflow, and suspected cause. If a live report already covers the same pattern, add only materially new evidence with `scout-edit-report`. Skip unchanged issues. Never create a second report for an unchanged issue.
## Investigate a bounded set
1. Rank identified users by generated-call spend. Include both `$ai_generation` and `$ai_embedding` when calculating full cost totals. Exclude rows where `distinct_id = properties.$ai_trace_id` when treating `distinct_id` as a user.
2. Establish the population baseline for cost share, traces, cost per generation, tokens, cache behavior, errors, and retries.
3. Select at most three candidates whose behavior materially differs from both the population and their own baseline.
4. Break each candidate down by model, provider, span, workflow, feature, or another property that exists in the project. Use `read-data-schema` before grouping by custom dimensions.
5. Open representative traces before explaining the cause. Aggregates identify candidates. Traces establish whether the cause is a retry loop, context growth, output growth, model choice, missing caching, abuse, or a product bug.
Minimize personal data. Use the least identifying stable label available. Never include raw prompts, responses, or full person-property objects in a report.
Close without a report when the highest-spend users are consistent with expected volume and normal unit economics.
## Report only actionable patterns
A report-worthy finding must be extraordinary against a relevant baseline, material and recent, supported by representative traces, and actionable through code, prompts, model choice, caching, limits, configuration, or product behavior.
Group users with the same root cause into one report. Create no more than two reports per run. Search the Inbox again before writing. Edit a matching live report instead of creating a duplicate.
Title a new report `Unusual AI spend: <segment and cause>`. Include the comparison window, the minimum numbers needed to judge the change, the trace-backed cause or best next investigation, and one specific next action. Include direct trace links or IDs as evidence.
Do not report routine top spenders, expected launches or batch jobs, test traffic, one costly trace without a repeatable problem, or a known provider incident already covered elsewhere.
Finish with a short run summary covering what you checked, what you reported or updated, and what you ruled out.
```

Show full example

Fig. 4 – The scout itself. The file an agent follows on a schedule.

The custom scout for this already lives in PostHog. You just have to enable it, and it's set to run daily at 9:00 AM.

You'll need these to run this scout.

-   Required[AI Observability](/docs/ai-observability/installation.md) instrumented, so $ai\_generation events are arriving
-   RequiredGenerations attributed to a user, so spend can be grouped by person rather than by trace
-   RecommendedA property identifying the feature or workflow, so a cause can be located

[Add this scout](https://app.posthog.com/ai-observability/self-driving#template=costly-users)Opens PostHog with it prefilled. Review it, then hit Create.

Not set up yet? One command installs PostHog, connects GitHub, and turns on a default troop of scouts. That troop doesn't include this one – add it above once you're set up.

```
npx @posthog/wizard self-driving
```

[‹ Silent failures](/pocket-guides/self-driving/silent-failure-core-action.md)[All guides](/pocket-guides.md)p. 5 of 7[Dead ends ›](/pocket-guides/self-driving/pages-people-cant-leave.md)

signals-scout-ai-observability-costly-users[Add this scout](https://app.posthog.com/ai-observability/self-driving#template=costly-users)