Technical SEO · Crawl Budget
Server logs tell you what Search Console will not
Search Console reports a sample of what happened to your pages. Log files record every request a crawler ever made. On a large site the gap between the two is where the budget is being wasted.
FissionX · September 5, 2026 · 7 min read
What the logs add
Search Console tells you which pages ended up indexed and roughly how often the site was crawled. It does not tell you that 40 per cent of crawl requests last month went to filtered category URLs with three parameters, or that your most profitable product template has not been fetched since a deploy in June.
Raw access logs answer both questions directly, because every request is in there with a timestamp, a user agent, a status code and a response time.
Verify the bot before you trust the row
User agent strings are trivially spoofed, and a meaningful share of traffic claiming to be Googlebot is not. Before analysis, verify by reverse DNS lookup on the IP and a forward lookup back, or match against the published IP ranges the search engines maintain.
Skip this and your "crawl budget" analysis is partly a report on scrapers.
The four questions worth asking first
Most log analysis gets lost in the tooling. Start with a narrow set of questions and the value shows up in an afternoon.
- Which URL patterns absorb the most crawl requests, and are any of them pages you do not want indexed?
- Which templates have not been crawled at all in the last 30 days?
- What share of crawler requests return 3xx, 4xx or 5xx — and does that spike at particular times of day?
- How does crawler-observed response time compare to what your monitoring reports for real users?
Crawl waste is usually a handful of patterns
Faceted navigation, session identifiers, calendar pages, internal search results and endlessly paginated archives account for most of the waste we find. They are rarely a surprise once seen — they are simply invisible until somebody looks at the requests.
The fix is usually robots.txt rules for genuinely worthless patterns, plus removing the internal links that generate them, since a URL nothing links to is a URL nothing crawls.
Use it to time your releases
Logs also tell you how quickly a template change is picked up. Push a fix, then watch the fetch pattern for that template. If the crawl frequency for a section is monthly, expect your change to take that long to be reflected, and stop attributing the flat line to the fix being wrong.
This is the quiet benefit of log analysis: it replaces argument about what is happening with a record of what happened.