Missing Page Types: Gaps Keywords Can't Show

Full transcript

1,032 words · script draft, under review.

Intro

Your keyword gap list has two hundred rows. You start building pages, one keyword at a time. A guide here, a product page there.

Meanwhile, your three best competitors all have something you don't: a set of comparison pages. Fifty of them. Each ranking for dozens of small searches that never made it into your gap list, one by one.

A keyword list can't show you that. It shows searches, not the shape of a site.

In this episode, we find the page types your competitors have and you don't, from their URLs.

Why keywords can't show page types

A page type is a template: a category page, a product page, a buying guide, a comparison, a calculator, a page per location. Sites grow by repeating templates, not by writing one page per keyword.

When you look at keywords, a page type is spread out over hundreds of small searches, often with low or zero reported volume. Each one alone fails your filters. Together, they're a whole section of the site doing real work.

You only see it when you group by URL instead of by keyword. That's what this episode does. It's the second of the three gaps from episode one, and it's the one most gap analyses miss.

Step 1: collect the URLs

Start with the sitemap. Open the competitor's robots dot txt. Most sites list their sitemap there. Often it's a sitemap index: a file that points to several child sitemaps, one per section. That structure alone is a hint, because sites often split sitemaps by page type.

Download them all, politely. One request at a time, with a pause, and a user agent that says who you are. You're reading public files, but you're still a guest.

If there's no sitemap, you have two fallbacks. The ranking URLs from your export in episode thirty-eight, which you already paid for. Or a small crawl, which we'll set up in the next episode.

Store every URL in a table, with the domain and the date. If the sitemap has a last-modified date per URL, keep that too. It's not always honest, but a section where nothing has changed in two years tells you something.

Step 2: group URLs into patterns

Now turn thousands of URLs into a handful of patterns. Take the path, and replace the variable parts with placeholders. Numbers become an id. Long hyphenated words become a slug. Keep the first folder or two as they are.

So "compare, grinder x versus grinder y" becomes "compare, slug". A product with a number becomes "product, id". Then count URLs per pattern.

Most sites collapse into five to fifteen patterns. That's the site's skeleton. Then open three random URLs from every pattern and look at them. Five minutes per competitor. The pattern tells you how many pages exist. Looking tells you what the template actually is.

When you open them, take notes on the same few things every time. What does the title look like across the three samples? What's the main content: text, a table, a tool, a list of products? Which other pages does it link to? Three samples that look the same mean a real template. Three that look different mean the pattern is just a folder name.

Watch out for patterns that are really filters. Sort orders, colors, price ranges, page two, page three. They can inflate a pattern to tens of thousands of URLs that are the same category page in disguise. Group them under their parent, and don't count them as a page type.

Step 3: compare the inventories

Now label each pattern with a page type, and build one matrix. Page types down the side, sites across the top, the number of URLs in each cell. Made-up numbers for illustration.

Our demo shop has categories, products, a dozen buying guides and some blog posts. Competitors B and D, our core competitors, have the same, plus two things we don't: comparison pages, around fifty each, and brand pages, one per manufacturer. Competitor A, the review site, also has comparison pages, and a calculator that tells you which brewer size fits how many cups. And the marketplace has category pages far deeper than ours.

The empty cells in our column are the page-type gaps. Three of them here.

Read the matrix the other way too. Is there a page type you have, and nobody else does? That's not a gap, it's an advantage. Write it down, because a gap analysis that only lists weaknesses gives a skewed picture to whoever reads it.

Now the step that turns a guess into evidence. Join the ranked keyword export from episode thirty-eight on URL pattern. For each competitor page type, sum the keywords and the volume of the URLs that rank. That costs nothing, the data is already stored. And it tells you whether the comparison pages actually rank, or just exist.

The pitfalls

Three pitfalls.

Sitemaps lie, in both directions. They can list pages that are blocked or no longer exist, and leave out pages that rank well. Treat the sitemap as a draft of the inventory, and cross-check with the ranking URLs.

Count is not quality. Five thousand location pages can be five thousand thin pages that barely rank. That's exactly why the ranking keywords column matters. A page type with lots of URLs and few rankings isn't a gap. It's a warning.

And copying is not fitting. A page type only belongs on your site if it serves your visitors and your business model. That's the role label from episode thirty-six at work. A marketplace's deep filter pages may not fit a small shop at all.

The output: a page-type gap list

What you get is a short list of page types. For each: which competitors have it, how many URLs, how many keywords those pages rank for, whether it fits your model, and one example to study.

Each row is not one page. It's a template, and a template can be many pages. That's why page-type gaps are often bigger than keyword gaps, and why they'll score high when we size everything in episode forty-three.

Wrap-up

Group URLs, not keywords, and the skeleton of a site appears. The sitemap pattern script is linked in the description. Next episode: the structural gap, how your competitors link their pages together, and why depth matters. See you there.