> AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt # Find costly AI users – Self-driving pocket guide Canonical URL: https://posthog.com/docs/self-driving/pocket-guide/costly-ai-users ## On this page - [What makes a user expensive](#what-makes-a-user-expensive) - [Divide cost by use](#divide-cost-by-use) - [Averages hide expensive users](#averages-hide-expensive-users) - [Anatomy of a trace](#anatomy-of-a-trace) - [You are not going to do this every morning](#you-are-not-going-to-do-this-every-morning) Chapter 05 # Find costly AI users – Self-driving pocket guide Tokenmaxxing is so in right now. But tokenmaxxing and value-minning? Less cool. A token leaderboard shows you your heaviest users – it doesn't show you your most expensive ones. A costly user is the one that should worry you: same work, same impact, more money. ## What makes a user expensive An expensive user isn't one who uses your product often – it's one who costs more without generating more usage or value. What triggers the cost increase is how you tell the difference. **Something you deployed triggered the cost change.** A model swap or a broken cache prefix triggers this. This is something you can spot on your AI bill because everyone's cost moves on the same day. [Tell an AI cost spike from real growth](https://posthog.com/docs/self-driving/pocket-guide/ai-spend-that-jumps.md) is the chapter that covers this. **The user's input caused it.** A 40-page PDF or six-hour conversation. Nothing changed in your code, but what it costs to serve that user did. This is the tokenmaxxing, value-minning user – and where you're actually losing money. ## Divide cost by use Your top spender is at $400 this month. Is that good or bad? You probably have no idea because spend is two numbers multiplied together: how much someone used your product, and what each use cost you. To figure it out, you should divide cost by use. PostHog already has the data you need to do this calculation – every AI observability event carries `$ai_trace_id`, and one [trace](https://posthog.com/docs/ai-observability/traces.md) is one full interaction: - The model call - Retrieval - Tool calls So you already have a denominator to divide by, and that $400 splits into two very different stories. In Fig. 1, two users have the same spend, but User A is the one using your AI product efficiently. Monthly spend = Traces×Cost per trace | | Monthly spend | Traces | Cost per trace | | | --- | --- | --- | --- | --- | | User A | $400 | 10,000 | $0.04 | Growth | | User B | $400 | 3,000 | $0.13 | Regression | Fig. 1 – Two users, the same monthly spend, and only one of them is a problem. The bar is cost per trace. This approach works well for traces. If you're building a multi-turn experience like a chatbot, use `$ai_session_id` instead. It groups traces into a whole conversation, and cost per session is the number that matters. ## Averages hide expensive users Watching cost per trace across your whole product works well for regressions that hit everyone, like a bad deploy or a model swap. This doesn't reveal edge cases and outliers. A retry loop that only fires on documents over 8k tokens hits the handful of accounts that send long documents. Average that in with thousands of healthy traces and it disappears. So one comparison isn't enough. You need two: how a user's cost per trace compares to everyone else's, and how it compares to their own last week. A user who doubled against themselves is worth a look even if they're nowhere near the top of your bill. ## Anatomy of a trace Back to User B in Fig. 1: cost per trace up 3x, routed through the same workflow. You know where the spend is going. You just don't know why yet. So you open User B's trace, and Fig. 2 is what you find. $ai\_tracedocument\_qa200 OK · 14.2s · $0.131$ai\_spanload\_document9,200 tokens in$ai\_generationextract\_fieldsstop: max\_tokens · $0.0241$ai\_generationextract\_fields · retry 1stop: max\_tokens · $0.024$ai\_generationextract\_fields · retry 2stop: max\_tokens · $0.025$ai\_generationextract\_fields · retry 3stop: max\_tokens · $0.024$ai\_generationanswer\_questionstop: end\_turn · $0.0342 Fig. 2 – One request from User B. Four attempts at the extraction step, every one of them billed, and one answer at the end. There's 5 [generations](https://posthog.com/docs/ai-observability/generations.md) in a request that should take one, against a population median of 1.3. *There's your retry loop.* Except a retry loop isn't automatically expensive. A 429 rejection bills nothing, because the provider never started generating. A generation that ran, produced tokens, and hit `max_tokens` bills in full. So check the stop reason before you blame the retries. Every attempt here stopped on `max_tokens`, and every one carries cost. That's the finding: the provider billed four times to produce nothing, because the extraction step retries on malformed output and never gives up loudly. Now you know what to fix. ## You are not going to do this every morning No one has time to rank users, work out the population baseline, compare every candidate against it, split by model and workflow, and examine their traces? And you need to do this every day. So hand it to a [scout](https://posthog.com/docs/self-driving/scouts.md). A scout runs on a schedule and you only hear about it when something needs your attention. When it finds something, you get a [report](https://posthog.com/docs/self-driving/reports.md) with the comparison window, the numbers you need to judge it, the cause it found in the traces, and what action to take. Fig. 3 Inbox 1 unread 1 Scout · costly users·3h ago ### One customer's cost per trace tripled after switching to the document workflow Four accounts moved from $0.04 to $0.13 cost per trace over two days while their trace volume held steady. All four route through the `document-qa` workflow, which retries the extraction step on malformed output without a cap. Traces show a median of 5 generations per request against a population median of 1.3, and the retried generations carry non-zero cost with a `max_tokens` stop reason. The provider billed every retry. 4 accounts, ~$180/day above their own baseline **Suggested action:** Cap the retry count on the `document-qa` extraction step and fail loudly instead of retrying on malformed output. An agent opens the pull request. You review and merge.