How to fix ai crawler not specified on Webflow
Your robots.txt doesn't explicitly allow several AI crawlers. They aren't blocked, but a number of AI engines act on explicit permission, so naming them removes any ambiguity.
Steps for Webflow
- Open Project Settings > SEO > Indexing > robots.txt.
- Remove the 'Disallow: /' rule under the AI bot's user-agent, or add an explicit 'User-agent: <bot>' then 'Allow: /'.
- Publish the site and verify at https://yourdomain.com/robots.txt.
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: ClaudeBot
Allow: /What is ai crawler not specified?
When a robots.txt exists but never mentions a given AI crawler (and no wildcard blocks it), that crawler's access is technically allowed but undeclared. Some AI engines are conservative and prefer an explicit allow before crawling and citing a site.
Explicitly allowing the major AI crawlers is a cheap, unambiguous signal that you want your store to be discoverable in AI answers. It costs nothing and closes the gap between 'not blocked' and 'clearly welcome'.
See the complete Ai crawler not specified guide for every platform and the full background.
Not sure if your Webflow store has this?
Run a free SEOLZ audit — we’ll find ai crawler not specified and every other issue across your whole site.
Scan my site free