Can the crawler get in?
AccessBefore anything else, the AI crawlers you want must receive a normal page. robots.txt is only half of it; firewalls and CDNs decide the rest.
- robots.txt allows the AI crawlers you chooseGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, each on purpose.
- Firewall and CDN rules tested with real user agentsA 403 from a bot rule looks exactly like being invisible.
- XML sitemap current, submitted to Google and BingSeveral assistants lean on Google or Bing results to find pages.
- No key page behind a login, pop-up wall or geo blockA crawler will not click “accept” to read your prices.
- Fast server response on the first requestCrawlers work on short timeouts; a slow origin gets a partial read.
ProofOn 14 Sep 2026 blueaquaint.com returned 403 to GPTBot, and to plain curl. The page is fine in a browser. It is on our fix list, and it is why this check comes first.

