← All checks
Fetch your site as GPTBot: bot reality test
Group A6 pointsrubric r1
We request the homepage twice, once with a Chrome user agent and once with the GPTBot user agent, then compare status, challenge markers and text similarity.
Why it matters
robots.txt only states intent. The real test is whether your server answers a bot request. Many WAFs return 403 or a "Just a moment" challenge page to AI user agents, so the crawler stores the challenge text instead of your content. The check awards points for a 2xx response, for no challenge page, and for text at least 80% the same as the browser version.
How to fix it
- In Cloudflare, check Security events for blocks on the
GPTBotuser agent and add an allow rule. - Turn off "Block AI bots" or bot fight mode for your public content paths.
- Allow the published OpenAI IP ranges instead of trusting the user agent string alone.
- Serve the same HTML to bots and browsers; do not return a JavaScript challenge to bots.
How to verify
Run curl -s -A "GPTBot/1.1" -o /dev/null -w "%{http_code}" https://example.com/ and expect 200.