Stop trusting User-Agent strings: a field guide to AI bot verification in 2026

The web your WAF rules were written for does not exist anymore.

Three years ago, "bot traffic" meant Googlebot, Bingbot, a long tail of SEO crawlers, and the scrapers you were already blocking. The verification problem was solved: Google did reverse DNS, Bing did reverse DNS, the long tail you didn't trust. Most ops teams had a User-Agent allowlist, a residential-proxy denylist, and that was the program.

Then OpenAI shipped GPTBot in August 2023. ChatGPT learned to browse. Perplexity launched. Anthropic shipped ClaudeBot. Google split Googlebot from Google-Extended so publishers could opt out of training without losing search. By the end of 2024, AI crawlers had gone from rounding error to a measurable share of origin traffic on most content sites; by mid-2025, multiple edge providers were reporting AI bot traffic in the double digits as a share of non-human volume on news and reference verticals.

The traffic is not the problem. The traffic is the symptom. The problem is that for the first time, "a bot read your page" has direct economic consequences (your content shows up in an answer instead of a search result, with no click) and direct legal consequences (training-data provenance, EU AI Act Article 53, California AB 2013, the CDSM Article 4 opt-out). What used to be a server-load question is now a licensing question, a copyright question, and in some jurisdictions a regulatory-evidence question.

And we are still grepping User-Agent strings.

Why the User-Agent stopped being enough

A few things happened at once.

The major LLM vendors split their crawlers into multiple personas. Anthropic alone runs three: ClaudeBot for training, Claude-User for human-driven browsing, Claude-SearchBot for search indexing. OpenAI runs four. Perplexity runs two. Each has different intent, different opt-out semantics, and (in the EU) different legal status under the TDM regime. A single User-Agent: AI* rule treats all of them the same and gets the wrong answer for two-thirds of the traffic.

The user-driven fetchers (Claude-User, ChatGPT-User, Perplexity-User) created a new category that nobody had a mental model for: a bot request whose underlying intent is a human reading your page. Block it and you break a user experience for someone who never visited your site directly; allow it and you cannot tell it apart from a training crawler that happens to have changed its UA. The category needs different routing rules than either the old "good bots" or the old "bad bots" buckets.

Spoofing got cheap. By 2024, every popular scraper framework shipped with rotating residential proxies and one-line User-Agent randomization. Putting ClaudeBot/1.0 in your headers takes one keystroke, and if the publisher's only check is the UA string, you have just bypassed their AI-bot policy entirely.

And the regulatory clock started ticking. EU AI Act Article 53 (applicable August 2025) requires GPAI providers to "identify and comply with" Article 4(3) opt-outs. California AB 2013 (operative January 2026) requires training-data documentation. None of these regulations directly mandate that publishers verify inbound bots, but all of them create a litigation environment where "we logged that ClaudeBot accessed our site despite our opt-out" is the dispositive fact, and "but the User-Agent could have been spoofed" is the dispositive defense.

If you are operating on UA strings alone, you have neither a working bot policy nor working evidence.

What the threat model actually is

There are three populations of traffic claiming to be AI bots, and they need to be distinguished:

1. The vendor's real crawler, hitting your origin from infrastructure the vendor controls.
2. Someone else (a scraper, a competitor, a bored teenager with a residential proxy) putting ClaudeBot/1.0 in the User-Agent header to bypass your bot rules.
3. The vendor's user-driven fetcher (Claude-User, ChatGPT-User, Perplexity-User), hitting your site because a human asked an LLM a question that required browsing.

(1) is a research/training crawler; you might block it, monetize it, honor robots.txt. (2) is hostile traffic and you want to drop it. (3) is the messy one, because the underlying intent is a human reading your page through an LLM, and blocking it breaks an experience for someone who never knew your site existed until ChatGPT mentioned it.

Treat all three as "Anthropic" and write one rule, and you will get the wrong answer for at least two of them.

The four verification mechanisms in production today

Across the major AI vendors there are exactly four ways a publisher can verify a bot is who it says it is. Every vendor uses some combination of these, and the combination matters.

Published IP allowlist. The vendor publishes a list of IP ranges (CIDRs) that its crawler operates from. You verify by checking whether the source IP of the request falls inside one of those ranges. Anthropic, OpenAI, Perplexity, Common Crawl, and Bytedance all publish IP lists in some form. Coverage and freshness vary; some vendors update frequently, others update when someone notices the doc is stale.

Reverse DNS, then forward-confirmed. You take the source IP, do a reverse DNS lookup, check that the hostname matches a vendor-controlled domain pattern, then resolve that hostname forward and confirm it returns the original IP. This is the technique Google has used for Googlebot for over a decade and it is still the most robust mechanism Google operates. Several other vendors use the same pattern.

HTTP Message Signatures (Web Bot Auth). The bot signs each request with a private key and includes a Signature header per RFC 9421. The vendor publishes the corresponding public key in a JSON Web Key Set at a well-known URL. You verify the signature against the published key. OpenAI rolled this out for its crawler family in late 2024; a handful of other vendors followed. This is the only mechanism that cryptographically binds a request to an identity, and it is the only one that survives an IP being reassigned to a different tenant of the same cloud provider.

Vendor-published verified-bots registry, signed. Cloudflare, Sigstore, and a few others maintain registries of known bot operators with signed identity claims. Useful for the long tail of crawlers whose operators are too small to run their own infrastructure but willing to register an identity.

There is no fifth mechanism.

What "covered" actually means

The vendors that matter to a publisher today (OpenAI, Anthropic, Google, Microsoft, Apple, Perplexity, Common Crawl, DuckDuckGo, Mistral, Yandex, Baidu) each publish something that lets you separate their real crawlers from imposters. Some publish more than one thing. Some publish what they say they publish; some are more aspirational. Picking the right verification mechanism per vendor, and updating the picker as vendors change posture, is the boring fiddly work that makes a verifier reliable.

What's not covered is a long tail of bots with meaningful traffic volume that publish nothing verifiable. For those the User-Agent is the only signal you have, and the User-Agent is not a signal. The honest answer is unverified, and you handle them with whatever bot-management product you already use.

The full per-vendor matrix, including which UAs are recognized and what tier each one falls into, is at /docs/operators. It updates as vendors change.

What goes wrong in practice

A few patterns from the last eighteen months, anonymized but real.

A news publisher had a WAF rule that allowed User-Agent: ClaudeBot and blocked everything else. They were scraped to death by a competitor running a Node script that hardcoded the Anthropic UA. Source IPs were a US residential proxy pool. The UA-only rule made them more vulnerable than no rule at all, because the rule signaled exactly which UA to spoof.

A SaaS company added an HTTP Message Signature check for GPTBot, did not update their CDN's caching layer, and accidentally cached a 401 response for a real OpenAI request. The 401 propagated to thousands of legitimate ChatGPT-User requests over the next two hours. The fix was a five-minute config change. The damage was a week of debugging why their content stopped showing up in ChatGPT answers.

A B2B publisher was negotiating a licensing deal with a major LLM vendor. The vendor claimed certain IPs were not theirs. The publisher's logs said otherwise. Three weeks of back-and-forth resolved when the vendor admitted that one of their staging fleets had been bleeding into production crawl traffic. The publisher only had a leg to stand on because they had been recording cryptographic verification results, not just UA strings, for the previous six months.

A research site implemented rDNS verification for Googlebot and got the regex wrong. They were silently rejecting half of legitimate Googlebot traffic for nine days. Their organic traffic dropped 14%. The fix was one character.

The pattern: verification is not hard, but it is fiddly, and the failure modes are quiet. You do not get an alert when you have configured your rDNS check incorrectly. You get a slow bleed of traffic you did not mean to drop, or a slow bleed of imposter traffic you did not mean to allow.

What we built

DutyRadar is a JSON API. You send a User-Agent and a source IP. You get back a verdict (one of verified_bot, unverified, or unknown), the operator slug, and the method that produced the verdict. Full request and response shape at /docs.

The hard part isn't the HTTP. It's the long tail of operational reality behind each verification surface: an IP that lands in two vendors' published lists, an rDNS lookup that times out at the wrong moment, a JSON Web Key Set that ships with a malformed key, a vendor that quietly rotates a CIDR on a Saturday and doesn't update their docs for a week. We notice and you don't. That is the product.

What this is not

DutyRadar is a positive identification system. It tells you, with high confidence, when a request is from a vendor it claims to be from. It does not tell you that a request is "definitely not a bot." A request that fails verification is just unverified; it might be an imposter, a legitimate small bot we do not yet have coverage for, or a human on a corporate proxy with a weird UA.

If your problem is "stop all scrapers regardless of who they claim to be," DutyRadar is the wrong product. You want a behavioral bot management product (Cloudflare Bot Management, DataDome, HUMAN, Akamai) and you want DutyRadar layered on top of it for the AI-specific verification. The two solve different problems and the right answer is usually both.

Where this goes

The vendors who matter are converging on HTTP Message Signatures. OpenAI shipped first; the IETF is actively standardizing the directory format. In two years, signature-based verification will be the baseline and IP allowlists will be the legacy fallback. We are building toward that endpoint.

In the meantime, the right architecture is to treat verification as a layer you call out to, not a thing you implement in your WAF. The IP lists change. The rDNS regexes have edge cases. The JWK Sets need cache management. None of that is interesting work. None of it is differentiating for your business. It is exactly the kind of thing that should be a service.

If you want to try it, the verifier on dutyradar.com takes a User-Agent and IP and returns a verdict in your browser. You can paste in real log lines. The free tier covers most small sites; sign up at dutyradar.com/signup when you want to wire it into your edge.

If you have a vendor we do not yet cover, or a verification mechanism we are missing, /docs/operators lists the current matrix. We add operators when there is real traffic asking for them.