SEOLZ

Ai crawler blocked

Quick win

Your robots.txt explicitly blocks one or more AI assistants from crawling your store, so those engines can't discover, index, or cite your products when shoppers ask for recommendations.

What it is

robots.txt is a small text file at your domain root that tells automated crawlers which parts of your site they may read. A 'Disallow: /' rule under an AI crawler's user-agent (for example GPTBot for ChatGPT, or PerplexityBot for Perplexity) tells that assistant it may not read any page on your site.

Why it matters

AI assistants increasingly answer shopping queries by crawling and synthesizing content from stores they can access. A blocked crawler removes your store from consideration entirely for that engine - competitors who allow access get recommended in your place. Unblocking is a one-line change with outsized upside for AI visibility.

How to fix it

  1. Open your robots.txt at https://yourdomain.com/robots.txt and find the 'User-agent' block for the named AI crawler.
  2. Remove the 'Disallow: /' line under that block, OR add an explicit allow group: 'User-agent: <bot>' on one line and 'Allow: /' on the next.
  3. If the crawler is only caught by a wildcard 'User-agent: *' with 'Disallow: /', add a dedicated allow group for each AI bot you want to permit so it overrides the wildcard.
  4. Keep any genuinely private paths (cart, checkout, account, admin) disallowed - you only want to open up your public catalog and content.
  5. Save, then reload https://yourdomain.com/robots.txt and confirm the bot no longer has 'Disallow: /'.
# Before (blocking ChatGPT's crawler):
User-agent: GPTBot
Disallow: /

# After (allowing it, while still protecting private paths):
User-agent: GPTBot
Allow: /
Disallow: /cart
Disallow: /checkout
Disallow: /account

Fix it on your platform

Pick your platform for the exact steps.

How to fix ai crawler blocked on Shopify
  1. From Shopify admin go to Online Store > Themes > (the three-dot menu on your live theme) > Edit code.
  2. Under the 'templates' folder click 'Add a new template' and choose 'robots.txt' (this creates robots.txt.liquid).
  3. Shopify's default template loops over rules; to allow an AI bot remove any block that adds 'Disallow: /' for it, or append an explicit allow group such as 'User-agent: GPTBot' then 'Allow: /'.
  4. Save. Visit https://yourstore.com/robots.txt and confirm the bot no longer has 'Disallow: /'.
Full Shopify guide →Official Shopify docs ↗
How to fix ai crawler blocked on WooCommerce
  1. If a physical robots.txt exists at your site root, edit it via SFTP/your host file manager.
  2. Otherwise edit it in Yoast SEO (Tools > File editor) or Rank Math (General Settings > Edit robots.txt).
  3. Remove the 'Disallow: /' line under the AI bot's 'User-agent' block, or add an explicit 'User-agent: <bot>' followed by 'Allow: /'.
  4. Save and load https://yoursite.com/robots.txt to verify.
Full WooCommerce guide →Official WooCommerce docs ↗
How to fix ai crawler blocked on BigCommerce
  1. Go to Channel Manager > your storefront > Customize robots.txt (or Settings > SEO depending on plan).
  2. Remove any 'Disallow: /' rule under the AI bot's user-agent, or add an explicit allow group for it.
  3. Save and verify at https://yourstore.com/robots.txt.
Full BigCommerce guide →Official BigCommerce docs ↗
How to fix ai crawler blocked on Squarespace
  1. Squarespace generates robots.txt automatically and does not allow direct editing.
  2. Make sure you have NOT enabled any setting that hides the site from search engines (Settings > Crawlers / 'Discourage search engines').
  3. Because per-bot robots rules aren't editable, lean on the other GEO levers (llms.txt, schema, sameAs) to maximise AI visibility.
Full Squarespace guide →Official Squarespace docs ↗
How to fix ai crawler blocked on Webflow
  1. Open Project Settings > SEO > Indexing > robots.txt.
  2. Remove the 'Disallow: /' rule under the AI bot's user-agent, or add an explicit 'User-agent: <bot>' then 'Allow: /'.
  3. Publish the site and verify at https://yourdomain.com/robots.txt.
Full Webflow guide →Official Webflow docs ↗

Does your site have this issue?

Run a free SEOLZ audit to find ai crawler blocked — and every other issue — across your whole site in minutes.

Scan my site free

Frequently asked questions

What is Ai crawler blocked?

robots.txt is a small text file at your domain root that tells automated crawlers which parts of your site they may read. A 'Disallow: /' rule under an AI crawler's user-agent (for example GPTBot for ChatGPT, or PerplexityBot for Perplexity) tells that assistant it may not read any page on your site.

Why does ai crawler blocked matter?

AI assistants increasingly answer shopping queries by crawling and synthesizing content from stores they can access. A blocked crawler removes your store from consideration entirely for that engine - competitors who allow access get recommended in your place. Unblocking is a one-line change with outsized upside for AI visibility.

How do I fix ai crawler blocked?

Your robots.txt explicitly blocks one or more AI assistants from crawling your store, so those engines can't discover, index, or cite your products when shoppers ask for recommendations.

Authoritative references

Related Generative Engine Optimization issues