Chapter 05

Find costly AI users

Tokenmaxxing is so in right now. But tokenmaxxing and value-minning? Less cool.

A token leaderboard shows you your heaviest users – it doesn't show you your most expensive ones. A costly user is the one that should worry you: same work, same impact, more money.

What makes a user expensive

An expensive user isn't one who uses your product often – it's one who costs more without generating more usage or value. What triggers the cost increase is how you tell the difference.

Something you deployed triggered the cost change. A model swap or a broken cache prefix triggers this. This is something you can spot on your AI bill because everyone's cost moves on the same day. Tell an AI cost spike from real growth is the chapter that covers this.

The user's input caused it. A 40-page PDF or six-hour conversation. Nothing changed in your code, but what it costs to serve that user did. This is the tokenmaxxing, value-minning user – and where you're actually losing money.

Divide cost by use

Your top spender is at $400 this month. Is that good or bad?

You probably have no idea because spend is two numbers multiplied together: how much someone used your product, and what each use cost you.

To figure it out, you should divide cost by use. PostHog already has the data you need to do this calculation – every AI observability event carries $ai_trace_id, and one trace is one full interaction:

• The model call
• Retrieval
• Tool calls

So you already have a denominator to divide by, and that $400 splits into two very different stories. In Fig. 1, two users have the same spend, but User A is the one using your AI product efficiently.

This approach works well for traces. If you're building a multi-turn experience like a chatbot, use $ai_session_id instead. It groups traces into a whole conversation, and cost per session is the number that matters.

Averages hide expensive users

Watching cost per trace across your whole product works well for regressions that hit everyone, like a bad deploy or a model swap.

This doesn't reveal edge cases and outliers. A retry loop that only fires on documents over 8k tokens hits the handful of accounts that send long documents. Average that in with thousands of healthy traces and it disappears.

So one comparison isn't enough. You need two: how a user's cost per trace compares to everyone else's, and how it compares to their own last week. A user who doubled against themselves is worth a look even if they're nowhere near the top of your bill.

Anatomy of a trace

Back to User B in Fig. 1: cost per trace up 3x, routed through the same workflow. You know where the spend is going. You just don't know why yet.

So you open User B's trace, and Fig. 2 is what you find.

There's 5 generations in a request that should take one, against a population median of 1.3. There's your retry loop.

Except a retry loop isn't automatically expensive. A 429 rejection bills nothing, because the provider never started generating. A generation that ran, produced tokens, and hit max_tokens bills in full. So check the stop reason before you blame the retries.

Every attempt here stopped on max_tokens, and every one carries cost. That's the finding: the provider billed four times to produce nothing, because the extraction step retries on malformed output and never gives up loudly.

Now you know what to fix.

You are not going to do this every morning

No one has time to rank users, work out the population baseline, compare every candidate against it, split by model and workflow, and examine their traces? And you need to do this every day.

So hand it to a scout. A scout runs on a schedule and you only hear about it when something needs your attention.

When it finds something, you get a report with the comparison window, the numbers you need to judge it, the cause it found in the traces, and what action to take. Fig. 3

Add the scout

The scout itself is just a file an agent follows on a schedule, and Fig. 4 is the one this guide is about.

---
name: signals-scout-ai-observability-costly-users
description: >
  Finds extraordinary user-level AI spend or usage, validates the cause in traces, and reports only
  patterns with a controllable next action.
scout-tags:
  - ai-observability
---

# AI observability costly or unusual users

Find extraordinary user-level cost or usage patterns in the most recent complete 24 hours. Compare them with the preceding 24 hours, the recent 7-day baseline, and the same weekday when traffic is seasonal. Never compare a complete period with a partial one.

A user ranking highly by spend is not a finding. Look for a material departure in unit economics or behavior with a controllable cause, validated in real traces. High volume with normal cost per trace may be healthy usage.

## Use the packaged analysis skills

Load these preinstalled skills through the runtime's packaged-skill mechanism when relevant:

- `analyzing-expensive-users`
- `exploring-llm-costs`
- `exploring-llm-traces`
- `querying-posthog-data`

These are packaged runtime skills, not project skill-store entries. Do not use `skill-list` or `skill-get` to load them.

## Avoid duplicate work

Read this Scout's last 14 days of run summaries with `scout-runs-list`, filtered by its exact `skill_name` and current `skill_version`. Retrieve relevant details with `scout-runs-retrieve`.

Search the scratchpad and recent Inbox reports for the user, segment, workflow, and suspected cause. If a live report already covers the same pattern, add only materially new evidence with `scout-edit-report`. Skip unchanged issues. Never create a second report for an unchanged issue.

## Investigate a bounded set

1. Rank identified users by generated-call spend. Include both `$ai_generation` and `$ai_embedding` when calculating full cost totals. Exclude rows where `distinct_id = properties.$ai_trace_id` when treating `distinct_id` as a user.
2. Establish the population baseline for cost share, traces, cost per generation, tokens, cache behavior, errors, and retries.
3. Select at most three candidates whose behavior materially differs from both the population and their own baseline.
4. Break each candidate down by model, provider, span, workflow, feature, or another property that exists in the project. Use `read-data-schema` before grouping by custom dimensions.
5. Open representative traces before explaining the cause. Aggregates identify candidates. Traces establish whether the cause is a retry loop, context growth, output growth, model choice, missing caching, abuse, or a product bug.

Minimize personal data. Use the least identifying stable label available. Never include raw prompts, responses, or full person-property objects in a report.

Close without a report when the highest-spend users are consistent with expected volume and normal unit economics.

## Report only actionable patterns

A report-worthy finding must be extraordinary against a relevant baseline, material and recent, supported by representative traces, and actionable through code, prompts, model choice, caching, limits, configuration, or product behavior.

Group users with the same root cause into one report. Create no more than two reports per run. Search the Inbox again before writing. Edit a matching live report instead of creating a duplicate.

Title a new report `Unusual AI spend: <segment and cause>`. Include the comparison window, the minimum numbers needed to judge the change, the trace-backed cause or best next investigation, and one specific next action. Include direct trace links or IDs as evidence.

Do not report routine top spenders, expected launches or batch jobs, test traffic, one costly trace without a repeatable problem, or a known provider incident already covered elsewhere.

Finish with a short run summary covering what you checked, what you reported or updated, and what you ruled out.

The custom scout for this already lives in PostHog. You just have to enable it, and it's set to run daily at 9:00 AM.

npx @posthog/wizard self-driving