Cloudflare, July 1,2025
Message Signatures Are Now Part Of Our Verified Bots Program, Simplifying Bot Authentication
As a site owner, how do you know which bots to allow on your site, and which you'd like to block?
more →
Back Issues/Search Home → Archive → Popular →
All sections ›
Archive ›
Vol 328 · Issue 1
Cloudflare — Archive · Vol 328 Issue 1
4 articles that week
Cloudflare, July 1,2025
From Googlebot To GPTBot: Who's Crawling Your Site In 2025
Web crawlers are not new. The World Wide Web Wanderer debuted in 1993, though the first web search engines to truly use crawlers and indexers were JumpStation and WebCrawler.
more → Cloudflare, July 1,2025
Content Independence Day: No AI Crawl Without Compensation!
Almost 30 years ago, two graduate students at Stanford University - Larry Page and Sergey Brin - began working on a research project they called Backrub.
more → Cloudflare, July 1,2025
The Crawl Before The Fall... Of Referrals: Understanding AI's Impact On Content Providers
Content publishers welcomed crawlers and bots from search engines because they helped drive traffic to their sites.
more →