Test the intended policy on representative URLs
Define the purpose of each product, filter, private-account, script and staging rule before testing representative URLs. Keep crawling policy separate from authentication and search removal. Narrow cases should prove that intended public content remains accessible without treating robots text as a security boundary or a guarantee of index exclusion.
Worked example — illustrative data.
| URL class | Proposed policy | Verification case |
|---|---|---|
| Useful product | Allow relevant search crawling | /products/arc/ returns useful content |
| Private customer account | Require actual authentication | /account/ is protected independently |
| Arbitrary filter combination | Evaluate scoped crawl restriction | Confirm useful landing pages remain accessible |
| Required script resource | Avoid accidental broad block | Product rendering still works |
| Staging site | Use appropriate access protection | Do not rely solely on robots text |
Write the purpose of each rule before writing syntax. Compare current provider documentation with the actual hostname, crawler token and path patterns. Test representative allowed and disallowed URLs, including an edge case that resembles a protected pattern. Keep the old and new intended policies in a migration review so staging restrictions are not copied into production unnoticed. Inspect actual responses and content as well as the rule file.
Read the page instruction before blocking the path
For a URL proposed for a crawl restriction, record its current page-level instruction and the reason for the change. If the intended outcome is removal from search, check the provider’s documented indexing method rather than assuming a crawl block produces it. Test a useful product, an unwanted filter and a required resource as separate cases after an approved rule change.
A crawl block can conceal a page-level noindex
Blocking crawling can prevent a crawler from seeing a page-level noindex instruction. If the goal is removal from search, choose a supported approach with that interaction understood instead of combining controls blindly.
Approve only the narrow rule set that serves the stated purpose and passes the URL cases. This table is a policy rehearsal, not a deployable robots file or proof of access.
Continue with Guides for stores, software teams and local businesses, Technical SEO services and Which store filters should Google index?.
For robots.txt and its limits, see Google robots.txt documentation (checked November 2026).