Group requests before choosing a crawl hypothesis
Group requests by useful product, filter space, old path and required resource, then verify the suspected discovery problem against actual response content. Treat request counts as evidence only within the log’s time, edge and origin coverage. A user-agent string alone does not authenticate a crawler or explain search visibility.
Worked example — illustrative data.
| Request specimen | Response/count specimen | Investigation |
|---|---|---|
| /products/arc/ | 200; 12 requests | Is useful content returned? |
| /lamps/?color=brass&size=tiny | 200; 400 requests | Repeated filter space; assess crawl purpose |
| /old/arc.html | 404; 25 requests | Known old URL or genuine removal? |
| /assets/product.js | 403; 10 requests | Could blocked resource affect rendering? |
These numbers are synthetic examples, not store measurements. Obtain a bounded permissioned log export and record time zone, retention window and edge/origin scope. Group by URL pattern, status and verified crawler where appropriate. Compare the suspicious pattern with catalog, linking and rendering evidence before changing robots rules or infrastructure. Inspect whether caching or a CDN means the log omits some requests. Keep sensitive query values out of the working report.
Check which requests the log can actually contain
Ask the hosting owner whether the export covers edge requests, origin requests or both, and record the time zone and retention window. For the sample’s absent product request, check whether cache hits could bypass the chosen log. Verify claimed Google requests through the documented method before labeling a crawler segment. A gap outside collection coverage remains unknown.
Decision: Prioritize a specific discovery or response hypothesis that can be checked. Repeat the review after the targeted change and keep crawl behavior distinct from search visibility.
A log gap may reflect incomplete collection coverage: A successful response can contain an error page or challenge, while a missing request may reflect incomplete log coverage. Do not translate either condition directly into an indexing conclusion.
Does a high request count prove a URL matters to shoppers?
No. A request count describes activity in the captured log, not customer demand or revenue. Combine it with the URL’s catalog purpose, internal links and available search evidence before prioritizing a change.
Continue with Guides for stores, software teams and local businesses, Technical SEO services and Check whether AI search crawlers can reach your pages.
For verifying requests that claim to be from Google, see Google crawler verification, checked 4 November 2026.