AI Crawl Blocker – Block AI Crawlers, Bots & Scrapers

By Jumedeen khan

10+ active installationsVersion 2.0.1

Block AI crawlers, scrapers and AI bots with robots.txt rules, and HTTP 403 blocking and protect your content.

Description

AI Crawl Blocker keeps AI crawlers and scrapers away from your content. It recognizes the crawlers behind the biggest names in AI and search, lets you decide which ones to allow and which to block, and shows you what each one is doing on your site.

You control how each bot is handled: asked to leave politely through a robots.txt Disallow rule, or turned away with an enforced HTTP 403 response when it actually reaches WordPress. The two methods are independent, so you can use either one or both.

Blocks AI Bots and Crawlers

The plugin ships with a registry of known crawlers, including the ones site owners most often need to stop:

  • OpenAI – GPTBot, ChatGPT-User, OAI-SearchBot
  • Anthropic – ClaudeBot, Claude-Web, Claude-User, Claude-SearchBot
  • Google – Googlebot, Google-Extended, Google-CloudVertexBot
  • Meta – FacebookBot, Meta-ExternalAgent, Meta-WebIndexer
  • Perplexity – PerplexityBot, Perplexity-User
  • Amazon – Amazonbot, Amzn-SearchBot, Amzn-User
  • Apple – Applebot, Applebot-Extended
  • ByteDance – Bytespider, TikTokSpider
  • Microsoft – Bingbot
  • Others – CCBot, DuckDuckBot, AhrefsBot, SemrushBot, and hundreds more grouped by category

Every bot is listed with its vendor and category, so you can see what a crawler does before deciding how to treat it. If one you care about is missing, add your own User-Agent string.

The crawler registry

Known crawlers are grouped into seven categories:

  • AI Crawler – Bots that bulk-collect content for training and research, such as GPTBot, CCBot, ClaudeBot, and Bytespider.
  • AI Assistant – On-demand fetchers behind assistants and chat features, such as ChatGPT-User and Claude-User.
  • AI Search – Crawlers used by AI search and answer services.
  • Search Engine Crawler – Traditional crawlers such as Googlebot, Bingbot, and DuckDuckBot.
  • SEO Tool – Crawlers used by SEO analysis tools, including AhrefsBot and SemrushBot.
  • Archiver – Web archiving crawlers.
  • Other Crawler – Monitoring services and link-preview bots.

Only AI Crawlers are blocked by default. Search engines, SEO tools, archivers, and the rest stay off until you opt in, so turning this plugin on never changes your search visibility on its own.

How a selected bot is handled

  • robots.txt rules – Publishes a Disallow rule for each bot you pick. Well-behaved bots respect it, though not every crawler does.
  • Hard block (403) – Sends an HTTP 403 response to a selected crawler when it reaches WordPress, whether or not it reads robots.txt.

Both methods cover the WordPress front end as well as the REST API and XML-RPC, so a selected crawler cannot reach your content through another endpoint.

AI usage signals

Beyond blocking, the plugin tells AI systems how your content may be used through a few standard signals, all on by default and individually toggleable from Settings:

  • A noai, noimageai, nosummary robots directive in the <meta name="robots"> tag of every page.
  • An X-Robots-Tag HTTP response header with the same directives.
  • A Content-Signal line in robots.txt that permits search indexing but declines use for AI training.
  • A W3C TDMRep manifest served at /.well-known/tdmrep.json.
  • A rights-reservation comment in robots.txt reflecting the EU Directive 2019/790.

Custom User-Agents and IP blocks

Crawlers that are not in the registry can be blocked by adding their User-Agent string. Up to 50 entries are supported, matched anywhere in the header and case-insensitively, and enforced with the same two methods as the built-in registry.

You can also block individual IPv4 or IPv6 addresses with a hard 403, independently of any User-Agent rules.

Works with caching and CDNs

Server-side detection only runs when a request reaches WordPress. A page served straight from a server cache, caching plugin, or CDN may never reach WordPress, so it cannot be classified there.

For that case, JavaScript Detection records known crawlers that load a cached page carrying the plugin’s lightweight detection script. It is enabled by default and can be switched off from Settings.

This is a monitoring feature rather than an enforced block: a cached page is already served, but the activity is logged so you know a blocked crawler ignored your rules.

Dashboard and Crawler Logs

The Dashboard summarizes crawler activity for the date range you choose: total, blocked, allowed, and unknown requests, the number of distinct crawlers seen, activity trends, a breakdown by crawler and category, and the pages receiving the most blocked requests.

The Crawler Logs screen is a searchable, filterable, paginated history of every classified request, each recording the time, crawler, path, action, and user agent. Logging is on by default and can be turned off, and you choose how long entries are kept.

Settings, export, and uninstall

Settings can be exported to a JSON file and imported on another site, which is handy for backups or moving between installations.

You can also reset everything to the recommended defaults and choose whether settings, logs, and statistics are kept or removed when the plugin is deleted.

The plugin stores everything in your own WordPress database and makes no outbound requests to third-party services.

Installation

  1. Upload the bravo-crawl-hunter folder to /wp-content/plugins/, or install the plugin through the WordPress plugins screen.
  2. Activate the plugin.
  3. Open AI Crawl Blocker from the WordPress admin menu.
  4. Review the crawler settings and choose which bots to block or allow.

FAQ

Does robots.txt block every bot?

No. robots.txt only provides instructions, and a crawler is free to ignore them. Enable Hard block (403) under Settings for a response that is enforced whenever a selected crawler reaches WordPress.

Will this block search engines like Googlebot or Bingbot?

Not by default. Search engine crawlers and archivers are part of the registry but stay unblocked until you select them yourself under Settings → Bot Protection.

Does it affect search rankings?

Only if you tell it to. AI crawlers are blocked by default, but Googlebot, Bingbot, and similar search visitors remain enabled unless you block them explicitly.

I use a caching plugin, server cache, or CDN. Will the plugin still work?

Server-side detection works whenever a request reaches WordPress. For pages served entirely from a cache, the JavaScript Detection feature (on by default) records known crawlers that load the page, so their activity still appears in your logs and dashboard.

Where can I see what crawlers have been doing?

The Dashboard gives you the overview, and Crawler Logs lets you search and filter individual classified requests.

Does logging slow the site or grow the database without limit?

Logs are pruned automatically against the retention window you set (30, 60, or 90 days, or keep them forever), so older entries are removed once they age past it. Logging can also be turned off entirely while the dashboard totals keep updating.

Does the plugin send data to any external service?

No. Settings, logs, and statistics are stored in your own WordPress database, and the plugin makes no outbound requests to third-party services.

Can I block a bot that is not in the registry?

Yes. Add a custom User-Agent string under Settings. Custom entries can be blocked with the same two methods as the built-in registry.

Screenshots

Changelog

2.0.1

  • Settings saved via one sanitized AJAX path; imports too.
  • “Today” shows the last 24 hours.
  • Hourly chart labels respect the site timezone.
  • All-time trend compares the full record to the prior window.
  • Removed the unused options.php submit path.

2.0.0

  • Block specific IPv4/IPv6 addresses with a 403.
  • AI usage signals: Content-Signal, X-Robots-Tag, meta directives, TDMRep, EU rights.
  • CSV export of crawl activity.
  • Compliance monitor: spot blocked bots served from a cache.
  • Chart.js charts, light and dark ready.
  • New AI bots blocked by default automatically.
  • Registry expanded to over 200 AI crawlers.
  • Searchable, filterable AI Bots Protection table.
  • Compact robots.txt with one Disallow group.
  • Cache flushing for more caching plugins.
  • Options moved to a bravo_ch_ prefix; data migrates automatically.

1.0.1

  • Added the optional noai, noimageai robots meta directives.
  • Updated plugin documentation and readme details.

1.0.0

  • Initial release.