DutyRadar docs · get a key

Common Crawl bot verification

Common Crawl publishes an IP allow-list for CCBot, which powers the WARC archives most major LLMs train on. Blocking CCBot removes a site from future open-source training corpora; that is a strategic call, not a technical one.

Bots covered

CCBot Training crawler ◑ IP-verified

Sample request

curl -X POST https://api.dutyradar.com/v1/classify \
  -H "Authorization: Bearer dr_live_..." \
  -H "Content-Type: application/json" \
  -d '{"user_agent":"CCBot/2.0"}'

Sample response

{
  "data": {
    "verdict": "verified_bot",
    "operator": "common_crawl",
    "bot": {
      "name": "CCBot"
    },
    "category": "training_crawler"
  },
  "error": null
}

Try it now

Free tier: 500 verifications / month.

Sign up in 30 seconds. We email you a verification link, you mint your first dr_live_… key from the dashboard, and you're calling /v1/classify in under a minute.