Ai crawler not specified

Quick win

Your robots.txt doesn't explicitly allow several AI crawlers. They aren't blocked, but a number of AI engines act on explicit permission, so naming them removes any ambiguity.

What it is

When a robots.txt exists but never mentions a given AI crawler (and no wildcard blocks it), that crawler's access is technically allowed but undeclared. Some AI engines are conservative and prefer an explicit allow before crawling and citing a site.

Why it matters

Explicitly allowing the major AI crawlers is a cheap, unambiguous signal that you want your store to be discoverable in AI answers. It costs nothing and closes the gap between 'not blocked' and 'clearly welcome'.

How to fix it

Open your robots.txt at https://yourdomain.com/robots.txt.
For each major AI crawler you want to welcome (GPTBot, ChatGPT-User, PerplexityBot, Google-Extended, ClaudeBot, Amazonbot, Applebot-Extended), add an explicit group: 'User-agent: <bot>' then 'Allow: /'.
Keep private paths (cart, checkout, account, admin) disallowed within each group if you wish.
Save and reload the file to confirm the new groups are present.

User-agent: GPTBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: ClaudeBot
Allow: /

Fix it on your platform

Pick your platform for the exact steps.

How to fix ai crawler not specified on Shopify

From Shopify admin go to Online Store > Themes > (the three-dot menu on your live theme) > Edit code.
Under the 'templates' folder click 'Add a new template' and choose 'robots.txt' (this creates robots.txt.liquid).
Shopify's default template loops over rules; to allow an AI bot remove any block that adds 'Disallow: /' for it, or append an explicit allow group such as 'User-agent: GPTBot' then 'Allow: /'.
Save. Visit https://yourstore.com/robots.txt and confirm the bot no longer has 'Disallow: /'.

Full Shopify guide →Official Shopify docs ↗

How to fix ai crawler not specified on WooCommerce

If a physical robots.txt exists at your site root, edit it via SFTP/your host file manager.
Otherwise edit it in Yoast SEO (Tools > File editor) or Rank Math (General Settings > Edit robots.txt).
Remove the 'Disallow: /' line under the AI bot's 'User-agent' block, or add an explicit 'User-agent: <bot>' followed by 'Allow: /'.
Save and load https://yoursite.com/robots.txt to verify.

Full WooCommerce guide →Official WooCommerce docs ↗

How to fix ai crawler not specified on BigCommerce

Go to Channel Manager > your storefront > Customize robots.txt (or Settings > SEO depending on plan).
Remove any 'Disallow: /' rule under the AI bot's user-agent, or add an explicit allow group for it.
Save and verify at https://yourstore.com/robots.txt.

Full BigCommerce guide →Official BigCommerce docs ↗

How to fix ai crawler not specified on Squarespace

Squarespace generates robots.txt automatically and does not allow direct editing.
Make sure you have NOT enabled any setting that hides the site from search engines (Settings > Crawlers / 'Discourage search engines').
Because per-bot robots rules aren't editable, lean on the other GEO levers (llms.txt, schema, sameAs) to maximise AI visibility.

Full Squarespace guide →Official Squarespace docs ↗

How to fix ai crawler not specified on Webflow

Open Project Settings > SEO > Indexing > robots.txt.
Remove the 'Disallow: /' rule under the AI bot's user-agent, or add an explicit 'User-agent: <bot>' then 'Allow: /'.
Publish the site and verify at https://yourdomain.com/robots.txt.

Full Webflow guide →Official Webflow docs ↗

Does your site have this issue?

Run a free SEOLZ audit to find ai crawler not specified — and every other issue — across your whole site in minutes.

Scan my site free

Frequently asked questions

What is Ai crawler not specified?

Why does ai crawler not specified matter?

How do I fix ai crawler not specified?