How to fix ai crawler not specified on WooCommerce
Your robots.txt doesn't explicitly allow several AI crawlers. They aren't blocked, but a number of AI engines act on explicit permission, so naming them removes any ambiguity.
Steps for WooCommerce
- If a physical robots.txt exists at your site root, edit it via SFTP/your host file manager.
- Otherwise edit it in Yoast SEO (Tools > File editor) or Rank Math (General Settings > Edit robots.txt).
- Remove the 'Disallow: /' line under the AI bot's 'User-agent' block, or add an explicit 'User-agent: <bot>' followed by 'Allow: /'.
- Save and load https://yoursite.com/robots.txt to verify.
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: ClaudeBot
Allow: /What is ai crawler not specified?
When a robots.txt exists but never mentions a given AI crawler (and no wildcard blocks it), that crawler's access is technically allowed but undeclared. Some AI engines are conservative and prefer an explicit allow before crawling and citing a site.
Explicitly allowing the major AI crawlers is a cheap, unambiguous signal that you want your store to be discoverable in AI answers. It costs nothing and closes the gap between 'not blocked' and 'clearly welcome'.
See the complete Ai crawler not specified guide for every platform and the full background.
Not sure if your WooCommerce store has this?
Run a free SEOLZ audit — we’ll find ai crawler not specified and every other issue across your whole site.
Scan my site free