Reading Your Own Access Logs, and the Two-Phase Check for a Real Googlebot

Analytics platforms disagree with each other, and every one of them is subject to the same limitation: a visitor with a script blocker is invisible. Usually that is a few percent. Sometimes it is a lot.

Access logs have no such blind spot. Every request that reaches your origin is in the log, because the log is written by the thing that served it. The cost is that logs record requests, not people — and turning one into the other is a modelling exercise with known error.

The format, field by field

The Apache documentation defines the common format as a specific string, which is the clearest way to learn the fields:

LogFormat "%h %l %u %t \"%r\" %>s %b" common

Field	Meaning	Notes
%h	Client IP	The field most analysis uses
%l	RFC 1413 identity	Documented as effectively unreliable
%u	Authenticated user	Empty on public sites
%t	Timestamp	[day/month/year:hour:minute:second zone]
%r	Request line	Method, path, protocol
%>s	Status code
%b	Bytes sent	Excludes response headers

The combined format adds two more:

\"%{Referer}i\" \"%{User-agent}i\"

Field	Meaning	Notes
Referer	The page that linked here	Often empty, and increasingly so
User agent	The client’s self-description	Trivially forged

Two documented caveats about specific fields. %l is the ident field, and Apache’s documentation notes it is “almost never” populated and should not be relied on. And on the client IP, the documentation makes a point that matters for interpretation: “The IP address reported here is not necessarily the address of the machine at which the user is sitting.” Proxies, CDNs and carriers all break the naive assumption.

%b excluding headers is worth internalising too. Your log’s byte count understates real transfer, sometimes substantially, because headers are not counted.

Verifying a crawler, properly

The user agent field is a string the client writes about itself. Anyone can write Googlebot. Which means crawler traffic in your log is a mixture of the real thing and anything claiming to be it — and the proportion of impostors varies enormously by site.

The documented verification method has two phases, and both are required:

1. Reverse DNS the IP from your log. The resulting hostname must be in one of Google’s documented domains — googlebot.com, google.com, or googleusercontent.com.
2. Forward DNS that hostname back. The resolved IP must match the original IP from your log.

Step two is the part people skip, and it is the part that makes the check work. Anyone can control reverse DNS for their own IP range, so step one alone proves nothing. What step two proves is that whoever controls that hostname has also published DNS pointing back at the same address — which is what an impostor cannot arrange for someone else’s IP.

# Phase 1: reverse lookup
dig -x 66.249.66.1 +short
# expect something like crawl-66-249-66-1.googlebot.com

# Phase 2: forward lookup on that name
dig crawl-66-249-66-1.googlebot.com +short
# must return 66.249.66.1

For bulk analysis, Google publishes machine-readable IP range lists for its crawlers, including separate files for common crawlers, special crawlers, and user-triggered fetchers. Matching against those lists is the automated version of the same idea, and note the documented reverse-DNS hostname patterns — crawl prefixes for the main crawler, and rate-limited-proxy prefixes for Google’s proxied fetches. A hostname that looks similar but does not match the documented pattern is not a Google crawler, however plausible it reads.

The platform distinction that trips people up

Three different things get conflated, and they are genuinely different products.

Beacon-based analytics is JavaScript that runs in the visitor’s browser. One major platform’s documented implementation loads a script from a static host and posts data to a path on your own domain. It is subject to script blockers and it does not see requests that never execute JavaScript.

Log-based analytics is derived from HTTP logs your origin produced. Nothing runs in the browser, so nothing can be blocked — but you also cannot distinguish a person from a bot without modelling.

Log delivery services export your logs to somewhere you can query. On one major CDN, the log-push product documents two constraints that decide whether it is usable: it “cannot backfill historical data” — meaning anything generated while it was not configured is permanently gone — and its availability is documented by plan, with the free, pro and business tiers not included.

That third constraint is the practical one. A log-delivery product that requires an enterprise plan is not a small-site tool, and the beacon-based product on the same platform is a different thing entirely. They share a vendor and nothing else.

What log analysis is good for

Bot traffic accounting. With proper verification, you can measure what fraction of your requests are crawlers. For a new site, that fraction is frequently the majority, which reframes every “growth” claim you might make.

Crawl behaviour. Which URLs is the crawler fetching, how often, and in what order? This is the only place to see it from the server side, and it explains a lot of otherwise confusing indexing behaviour.

Error rates. The distribution of status codes across your log is a direct measure of how often real requests fail, with no sampling and no script blocker bias.

The requests that never ran JavaScript. Download counts, feed fetches, direct image hotlinks — all invisible to beacon analytics, all present in logs.

What they are bad at

Unique visitors. IP addresses are shared by households, offices, carriers, and — critically — not stable over time. Any count you produce is a model with assumptions, not a measurement.

Anything requiring persistence. With no cookie and no client-side identifier, you cannot follow a session across an IP change.

Referrer data. Modern browser defaults and privacy policies strip or fake referrers, so the field is far less informative than it was.

The privacy dimension, stated carefully

Logs contain IP addresses, which are widely treated as personal data under European data protection law. I could not verify this from a primary source in this pass — the relevant regulator pages were unreachable — so treat the following as framing rather than citation: if you retain logs containing IP addresses, you have a data-retention question, and the standard answer involves minimising both what you store and how long you keep it.

The practical measures, all of which also happen to make the logs easier to work with:

• Aggregate early. Derive country, status class, or path counts, then delete the raw lines.
• Truncate or hash the IP if you only need approximate uniqueness.
• Set a retention period and enforce it automatically rather than by intention.
• Say so in your privacy page. If your privacy policy describes a beacon-based analytics product and you are also keeping raw logs, the policy is incomplete.

A first analysis, in order

1. Total requests, by status class. What fraction of everything is 4xx and 5xx? This is your baseline.
2. Verified crawler share. Run the two-phase check on the top IPs, or match against the published ranges. Know how much of your traffic is crawlers.
3. Top paths. What are crawlers fetching versus what are people fetching?
4. 404 list, sorted by count. The most-requested missing URL is a free content idea or a redirect you should write.
5. Crawl frequency by path. Are your new pages being fetched at all? That is the earliest signal about whether indexing is working.

Step four is the one with immediate practical value on a content site. If a URL is being requested hundreds of times and returning 404, something out there links to it — and the log just told you what to publish or redirect.