Block AI Training Crawlers in robots.txt
Adds rules to your virtual robots.txt that ask AI training bots such as GPTBot, ClaudeBot, CCBot and Google-Extended not to crawl your site.
Opens WPIDE on
Code
// AI crawler user agents to disallow.
$wpide_ai_bots = array( 'GPTBot', 'ChatGPT-User', 'ClaudeBot', 'anthropic-ai', 'CCBot', 'Google-Extended', 'Applebot-Extended', 'PerplexityBot', 'Bytespider', 'meta-externalagent' );
add_filter( 'robots_txt', function ( $output, $public ) use ( $wpide_ai_bots ) {
if ( ! $public ) {
return $output;
}
$rules = "\n# Block AI training crawlers\n";
foreach ( $wpide_ai_bots as $bot ) {
$rules .= 'User-agent: ' . $bot . "\nDisallow: /\n\n";
}
return rtrim( $output ) . "\n" . $rules;
}, 20, 2 );Notes
robots.txt is a request, not an enforcement. A physical robots.txt file in the site root overrides the virtual one. Edit the list to allow bots you want.
How to add this snippet
- Install WPIDE on your WordPress site (it is free) and open Code Snippets.
- Use snippet above adds it from the library in one click. Or copy the code into a new snippet with the same code type.
- It is added inactive: review the code and settings, then switch it on.