The definition
An AI crawler fetches web content for AI systems: training corpora, retrieval indexes, or live browsing on behalf of a user query. Different operators run different crawlers with different purposes, and they can be allowed or blocked separately.
The set changes as operators launch, rename and retire them.
Why this is a decision, not a default
Blocking a crawler means your content is not used to build answers, which also means you are far less likely to be named in them. For a publisher whose product is content that can make sense. For a local service business whose product is the job, being cited is the goal.
Most robots.txt files were never decided. They were inherited from a theme, a plugin, or a developer, years ago.
Check before assuming
We regularly find sites blocking crawlers their owners would never have chosen to block, and CDN or firewall rules blocking access independently of robots.txt. Both are invisible from inside a browser.
Check it yourself
Open yourdomain.com/robots.txt in a browser and read it. If you do not recognise a directive, you did not decide it.
Then check whether your CDN or firewall is blocking crawler traffic independently — that one is invisible in robots.txt and common.
See where you actually stand
A free Visibility Check runs twelve real buying questions for your category across three surfaces.
Get my free Visibility Check