Crawl vs Index vs Rank: Reading Crawl and Index Signals

Full transcript

1,279 words · script draft, under review.

Intro

You publish a large batch of pages. Weeks later, only a fraction of them show up in Search Console with any impressions at all. Is Google ignoring the site? Are the pages bad? Or is it just slow?

You can't answer that from the performance report. Impressions are the last step of a long chain. If the chain breaks earlier, the performance report just shows you silence.

In this episode we split that chain into three stages: crawl, index and rank. For each stage I'll show you which signal to read on one of my own sites, how far you can trust it, and the one measurement mistake I made that you can avoid.

The funnel: four numbers, not one

Think of it as a funnel with four levels. This one is real, from one of my own sites, example-city-stats.com. It's anonymized: the domain, the URLs and the search terms are replaced, so we stay on the principles and not on the keywords. The numbers are untouched.

At the top is your corpus: the pages you want Google to know about. Measure it as the unique URLs in your live sitemap. Not the sum of the sitemap entries in Search Console. If you submit a sitemap index, Search Console lists the index and every child file, and the index already counts the children. Add them up and you double your denominator. On my site, the corpus is just under fourteen thousand URLs.

Next level: crawled. Has Googlebot actually fetched the page? On my site, in the nine days my CDN keeps its log, verified Googlebot fetched about seventy-eight hundred of them, a bit more than half. Nine days is a short window, so read that as a pace, not a verdict. Then indexed. Did Google decide to keep it? And at the bottom: impressions. The page appears for real searches. That's what ranking means in practice.

The point of the funnel is that each level has a different cause when it's thin. Few pages crawled is a discovery or capacity problem. Crawled but not indexed is a quality or duplication judgement. Indexed with no impressions is a demand or ranking problem. Three problems, three different fixes. So you need a number at every level.

Crawl: the leading indicator

Start at the top. The best free crawl signal is the Crawl stats report in Search Console. It sits under Settings, not in the main menu, which is why many people never open it.

It shows how many requests Googlebot makes per day, how your server responded, and the purpose of each request. Purpose is the interesting one. Discovery means Google is fetching URLs it hasn't seen before. Refresh means it's re-checking pages it already knows.

That report is only in the interface, not in the API. So for my site I built the same view from the CDN side: requests per day from IP-verified Googlebot, and how many of the URLs were new. Eight weeks. On average about six thousand requests a day, swinging between roughly twelve hundred and thirteen thousand. And each week, somewhere between a fifth and just over two fifths of the fetched URLs were new.

In my experience, the crawl curve moves days to weeks before indexing does. That makes it the leading indicator. But be honest about what a chart can show. On this site, the index sample was a flat hundred percent every week, so there's nothing downstream for the crawl line to lead. If crawl requests are flat and discovery is near zero, nothing downstream will move either. Then stop waiting, and look at internal links and sitemaps.

Crawl behind a CDN: don't trust your server log

Here's the mistake I mentioned. If your site sits behind a CDN like Cloudflare, your own server log does not measure crawling. It measures cache misses. Most of Googlebot's requests are answered by the CDN and never reach your server. In one measurement on my sites, the origin saw about five percent of Googlebot's requests. Counting crawl from that log would have told me Google had almost lost interest. It hadn't.

The second trap is identifying Googlebot at all. Never trust the User-Agent string. Anyone can claim to be Googlebot. When I checked across my sites, roughly fifty-six percent of requests calling themselves Googlebot came from somewhere else.

The reliable method is to check the client IP against the ranges Google publishes for its crawlers. Behind a CDN, do that on the CDN's own analytics, where you see every request, not on the origin log. On screen is one day on example-city-stats.com in late August. About thirteen thousand requests from real Google addresses, and about three thousand that only claimed to be Googlebot. On this site the fakes came in waves: thousands a day on a handful of August days, under a hundred a day since September.

Index: sample, don't trust the sitemap count

Now the middle of the funnel. How many of your pages are indexed?

The tempting answer is the indexed figure attached to your sitemaps. Don't use it. In my experience it's unreliable, and it can sit unchanged for weeks while reality moves. Real index status comes from URL Inspection, or from searching for the exact URL in quotes.

URL Inspection has an API, but with a daily quota per property, so you can't inspect a large site every day. The practical answer is sampling.

Split your URLs into layers, by section or template. Pick a fixed number at random from each layer. Inspect them once a week. Multiply each layer's indexed share by its size, and you have an estimate of indexed pages, with a trend line. That's a stratified sample.

On my site, that's six sections, A to F, from fifty pages to about fifty-five hundred. A hundred and thirty-two URLs in the latest sample, and every one came back indexed. With a sample that size the margin is about four percentage points, so the honest reading is: above ninety-five percent.

Two rules make it trustworthy. First, the coverage state is reliable in aggregate, not for a single URL. In my measurements about three percent of "unknown to Google" answers were false. So never act on one URL's verdict without an exact-phrase search. Second, the endpoint can fail on a whole day at once. So in every run, inspect a site you know is well indexed as a control. If the control suddenly looks unindexed, the problem is the API, not your site.

Rank: indexed is not the finish line

The bottom level is the easy one to measure. Count the pages with at least one impression in Search Console over the last four weeks. Compare it with your indexed estimate.

On my site, about eleven thousand four hundred of the fourteen thousand had an impression, roughly four in five. About twenty-nine hundred got a click. So crawling and indexing aren't the problem here. This funnel narrows at the bottom. Either there's little demand for those pages, or they rank too low to be seen. That's where the rest of this series takes over: demand in the next module, and the results page after that.

Reading the funnel

So here's how to read it. Narrow at the top: Google isn't finding or fetching your pages. Check internal links, sitemaps and server responses, and watch discovery. Narrow in the middle: Google fetches the pages but chooses not to keep them. Look for thin or near-duplicate pages. Narrow at the bottom: the pages are in the index, but nobody sees them. That's demand and competition.

And re-read it every week. The shape over time tells you more than any snapshot.

Wrap-up

Crawl, index, rank: measure each level separately, and you'll know where to look. The funnel worksheet and the sampling script are linked in the description. Next episode: search volume, and where those numbers really come from. See you there.