Check rules, responses and useful content
Compare the intended crawler rule with the actual response body and useful page content. An allowed robots entry cannot prove that a security challenge, blocked resource or empty shell is absent. Record provider purpose and verified request context before deciding which narrow access layer needs investigation.
Worked example — illustrative data.
| Layer | Specimen condition | Verification |
|---|---|---|
| Robots rules | Search crawler allowed; training crawler disallowed | Match exact current provider token |
| Edge response | Product URL may receive a challenge | Inspect authorized logs and response behavior |
| Page content | Product description loads after scripts | Compare response and rendered text |
| Discovery | Product is linked from a collection | Check actual href and destination |
Start with the provider’s current crawler documentation and record the token and stated purpose. OpenAI documents OAI-SearchBot separately from GPTBot; do not assume one setting controls both uses. Inspect security and hosting behavior with the responsible owner rather than disabling protections broadly. A user-agent string alone is insufficient proof that a request came from the claimed provider. Keep access checks narrow and retain dates.
Test one public URL before changing a whole crawler policy
Choose a product page that should be public and write down the intended crawler purpose before reviewing access. Compare the returned title and product description with the expected page, not just the status. If a challenge appears, pass the exact URL, time and request context to the hosting owner.
Retest the same page after an approved narrow change and include an account page as an unaffected control.
A successful status can still return a challenge page
A page can return a successful status while showing only a challenge or empty application shell. Record the returned body as well as the status so a nominally successful response is not mistaken for useful access.
Change only the confirmed blocking layer after review. Treat successful access as a technical prerequisite, not a promise that the page will appear in answers.
Does changing robots.txt make private pages safe?
No. Keep account and other private content behind appropriate access controls. Crawler instructions govern requested crawling behavior; they are not authentication. Do not expose private material to make a public product URL easier to fetch.
Continue with Guides for stores, software teams and local businesses, Technical SEO services and Review store robots rules before a migration.
For the separate OpenAI crawler purposes and controls, see OpenAI crawler documentation (checked November 2026).