← All checks
Google-Extended and Applebot-Extended robots.txt check
Group A1 pointrubric r1
We test the path / against the Google-Extended and Applebot-Extended product tokens in your robots.txt.
Why it matters
These tokens do not crawl on their own. They tell Google and Apple whether pages they already fetched may feed their AI models. Disallowing them keeps your content out of Gemini and Apple Intelligence, so those assistants know less about you. It is a policy decision, but it is worth one point because many sites block them by accident with a blanket rule.
How to fix it
- Remove
Disallow: /from groups forGoogle-ExtendedandApplebot-Extended. - Do not add a blanket
User-agent: */Disallow: /that also catches these tokens. - To opt in explicitly, put
User-agent: Google-ExtendedandUser-agent: Applebot-Extendedabove oneAllow: /line. - If you must block them, keep the rule and accept the loss of the point.
How to verify
Re-scan this page; the evidence reads "neither blocked".