Introducing More Granular Controls for AI Bot Traffic
Akamai Technologies, Thursday, September 3rd, 2026
Akamai splits AI bot traffic into training crawlers, search crawlers and fetchers so sites can allow or block each type.
Akamai has updated its Bot Directory to classify AI bot traffic into three distinct categories: training crawlers, search crawlers, and fetchers/agents.
The change reflects the reality that not every AI bot visits a site for the same reason, so blanket allow or block rules are increasingly blunt.
Customers can now restrict content-scraping crawlers that harvest data for model training while still permitting AI search indexers and real-time user-triggered agents.
Akamai pairs the new visibility with its AI Brand Presence product so organizations can protect intellectual property without losing discoverability across AI platforms.