Allow AI crawlers to access your website
AI crawler bots are blocked by default - how to allow them for your website.
By default our servers block AI crawler bots with a global rule. We introduced it after AI bots repeatedly scraped customer websites so heavily that they caused high server load and slowed the websites down.
If you want AI crawlers to be able to read your website — for example so that AI assistants can find your content — you can switch the rule off for your website.
Allow AI crawlers
Add this rule to the web.config file in the wwwroot folder of your website:
<configuration>
<system.webServer>
<rewrite>
<rules>
<!-- Allow access to website for AI Bots -->
<remove name="HardRule: Block AI Bots" />
</rules>
</rewrite>
</system.webServer>
</configuration>
If your web.config already has a <rewrite><rules> section, add only the <remove ... /> line into it.
Important
When you publish a new version of your application, the web.config on the server may be replaced by the one from your project. Add the rule to the web.config in your project, otherwise it disappears with the next deployment.
Blocking other bots
The same web.config file lets you write your own URL Rewrite rules — for example to block an aggressive scraper by its user agent or address range before the request reaches your application.