Block bots in robots.txt

List of bots that can be blocked in robots.txt.

robots.txt is a text file in site’s root directory, specifically intended to communicate with web crawlers and bots.

BLEXBot Crawler

User-agent: BLEXBot
Disallow: /

Common Crawl (CCBot)

User-agent: CCBot
Disallow: /

Google AI

User-agent: Google-Extended
Disallow: /

OpenAI

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /