Managing bot and AI traffic
Contents
Bots, crawlers, and AI agents make up a growing share of web traffic, and not all of it is noise. PostHog lets you decide – per type of traffic – whether to welcome it, measure it, or keep it out of your numbers, and gives you a dedicated Bots tab in Web Analytics to watch it. This guide covers those decisions and where to act on them. For the underlying functions and properties, see bot and traffic detection.
Decide what to keep
The right move depends on the kind of automated traffic. A useful starting point:
| Traffic | Examples | Typical decision |
|---|---|---|
| AI agents | GPTBot, ClaudeBot, PerplexityBot, ChatGPT-User | Increasingly worth measuring – these visits are how AI tools find and cite your content. Often you want to see them, not drop them. |
| Search crawlers | Googlebot, Bingbot | Keep out of human metrics, but worth tracking for SEO. |
| SEO and monitoring tools | AhrefsBot, SemrushBot, Pingdom, UptimeRobot | Usually noise – exclude from analytics. |
| HTTP clients and headless browsers | curl, Python requests, Puppeteer, Selenium | Usually noise – exclude from analytics. |
You don't have to treat every bot the same. Because classification happens at query time, you can exclude monitoring tools from a dashboard while still breaking down AI agent traffic in a separate insight.
Three places to act
There are three points where you can deal with bots, from most aggressive to most flexible:
- Block at capture (client-side). The PostHog JavaScript SDK blocks known bots before they ever send an event. Lowest effort, but you lose the data entirely, and it only affects traffic that runs JavaScript. See blocking bots.
- Exclude at query time. Keep the events and filter them out where you don't want them – add an Is bot (
$virt_is_bot) isfalsefilter to an insight or dashboard. Flexible and reversible: the underlying data stays intact, so you can include or break down bot traffic whenever you want. - Capture server-side to measure bots. Most bots and AI agents never run JavaScript, so the SDK never sees them. To measure that traffic, forward your server logs as
$http_logevents – they carry the user agent and get classified the same way.
See bots in the Bots tab
The Bots tab in Web Analytics is a dedicated dashboard for automated traffic, scoped to detected bots so it stays separate from your human metrics. It shows:
- Bot requests over time – a trend of bot traffic, with breakdowns by crawler, category, host, and path.
- Crawlers – every bot that hit you in the selected period, with its category, request count, and when it was last seen.
- Most crawled paths – the pages bots request most, and how many different crawlers hit each one.
The tab reads $pageview, $screen, and $http_log events, so forwarded server logs show up alongside anything the JavaScript SDK captured. Classification is based on the user agent, and events without one are excluded from this tab – so when forwarding logs, make sure each event sets $raw_user_agent.
Note: The Bots tab is in alpha and rolling out gradually – it may not be enabled for your project yet. The
$virt_*properties and classification functions are available to everyone today.
Exclude bots from an insight or dashboard
In any Product Analytics insight or Web Analytics view, add a filter where Is bot equals false. Your visitor, session, and pageview counts then reflect human traffic only, without changing the stored data.
The property picker lists these under their display names rather than their raw keys, though searching for either works:
| Name in the UI | Property |
|---|---|
| Is bot | $virt_is_bot |
| Traffic type | $virt_traffic_type |
| Traffic category | $virt_traffic_category |
| Bot name | $virt_bot_name |
| Bot operator | $virt_bot_operator |
To exclude a narrower set – say, only automation and HTTP clients while keeping AI agents – filter on Traffic type or Traffic category instead.
See which bots are hitting you
- Right now – open the Live tab and check the bot traffic tile to see which bots are crawling you in real time.
- Over time – open the Bots tab for a ready-made dashboard, or build a trend filtered to Is bot is
trueand broken down by Bot name. See example queries for the SQL equivalent.
Track AI bot traffic
AI traffic is worth measuring on its own: these visits are how AI tools discover, index, and cite your content. PostHog classifies it into three categories:
- AI crawlers (
ai_crawler) – collect training data. Examples: GPTBot, ClaudeBot, Google-Extended. - AI search crawlers (
ai_search) – index your site for AI-powered search results. Examples: OAI-SearchBot, Claude-SearchBot. - AI assistants (
ai_assistant) – fetch a page because a person asked an assistant about it right then. Examples: ChatGPT-User, Claude-User, Perplexity-User.
To track them:
- Capture the traffic. Most AI bots don't run JavaScript, so the SDK never sees them. Forward your server or CDN logs as
$http_logevents – without this, you only measure the small slice of AI traffic that executes your snippet. - See who's reading your site. Open the Bots tab and switch the trend to the Category breakdown, or build a trend on the
$pageviewand$http_logevents (a$pageview-only trend misses the server-side traffic you set up in step 1), filtered to Traffic type equalsAI Agentand broken down by Bot name or Bot operator to compare OpenAI, Anthropic, Perplexity, and others. - See what they're reading. Check Most crawled paths in the Bots tab, or break the same trend down by
$pathname, to find which content AI tools hit most.
Assistant traffic deserves particular attention: each ai_assistant request has a person on the other end asking about your content right now, even though the request itself is automated. A rise in assistant traffic to a page means AI tools are reading it on users' behalf.
Related
- Bot and traffic detection – the classification functions and virtual properties
- Live – watch bot traffic in real time
- Troubleshooting – why your counts differ from other tools