Re-Measure After Shipping: Did the Fix Work?

Full transcript

1,070 words · script draft, under review.

Intro

Six weeks ago you rewrote the titles on four category pages. Today, clicks on those pages are up twenty percent. Made-up numbers, but a very real situation. Did the fix work?

Maybe. Or maybe it's the season. Maybe Google rolled out an update that lifted the whole site. Maybe one of the pages just got recrawled and indexed with a better snippet for reasons that have nothing to do with you.

Most before-and-after comparisons in SEO can't tell these apart. They show a line going up after a change, and everyone is happy until the line goes down again.

In this episode: how to set up a re-measurement before you ship, how long to wait, how to use control pages, and how to reach an honest verdict.

Before you ship: the change log

Re-measuring starts before you ship, not after.

For every change, I write one row in a change log. The date and time it went live. The exact URLs. What changed, in one sentence. The hypothesis: "a clearer title will raise the click-through rate at the same position." The one metric that decides it. The control pages. And the re-measure date, which we set back in episode forty-four.

Choosing the metric in advance is the important part. If you pick it afterwards, you'll find a metric that went up. There's always one.

I also store a snapshot of the results page for the main query on the day I ship. If the results page itself changes during the window, a new feature, a new competitor, I want to be able to see that.

The measurement window and indexing lag

A change doesn't exist for Google until Google has recrawled the page. So the clock starts at the recrawl, not at the publish. URL Inspection shows the last crawl date for a page. Check it, and start your window from there.

How long should the window be? These are rules of thumb, not laws. For a title or snippet change on an indexed page, two to four weeks after the recrawl. For a substantial content update, four to eight weeks. For brand new pages, you first need them crawled and indexed, as in episode seventeen, and that can take one to three months before rankings mean anything.

Then compare equal windows. Twenty-eight days after, against twenty-eight days before. Same number of each weekday, because many sites have a weekly rhythm. Leave out the days around the change itself. And remember that Search Console data arrives with a delay of a few days, so don't measure the last two or three days of the window too early.

If the topic is seasonal, a before-and-after comparison is close to useless on its own. Use the same weeks last year as well, like in episode thirteen.

Control pages

This is the single biggest improvement you can make to your measurement: a control group.

Pick pages that are similar to the ones you changed, same page type, similar traffic level, same part of the site, and that you did not touch. Measure them over exactly the same windows.

Here's the example shop. The four pages with new titles went up twenty percent. The eight similar category pages that weren't changed went up eighteen percent. So the effect of the titles is roughly two percentage points. Probably noise. Something else lifted the whole section, and the titles got credit for it.

Flip it around. If the changed pages went up twenty percent and the controls went down five, you have a much stronger case.

That's all there is to it. Compare the change in the changed group with the change in the control group. The difference is your estimate of the effect.

Which metric for which change

Use the metric that matches the hypothesis.

For a title rewrite, the decision metric is click-through rate at a comparable position. If the position moved a lot during the window, the click-through rate isn't comparable, and you need to say that. Remember that average position is weighted by impressions, as in episode eight.

For a content update, look at impressions and position on the page's queries. Clicks follow later.

For new pages, go down the funnel in order. Are they indexed? Use URL Inspection on a sample. Then impressions. Then position. Then clicks. Judging new pages on clicks after two weeks is the most common way to kill a good idea early.

And measure at page level first. Query-level data hides a share of the queries, as we covered in episode nine, so page totals are the more reliable before-and-after number.

The verdict

End with one of three verdicts, and write it into the change log next to the hypothesis.

It worked: the changed pages beat the controls on the metric you chose in advance. Then the change becomes a candidate to roll out to similar pages.

It didn't: no difference against the controls, or worse. Revert if it hurt, keep it if it's neutral and better for users.

Inconclusive: too little data, the results page changed, or the controls moved in a strange way. That's a valid answer. Extend the window or run a cleaner test. Small pages with a few clicks a week will often land here, and pretending otherwise is how fake wins end up in reports.

One more rule: one change per page per window. If you rewrite the title, add content and add internal links in the same week, you won't know which one did it.

Three traps in before-and-after

Three traps to watch for.

Moving the goalposts. The decision metric didn't move, so you report a different one that did. That's why the metric goes into the change log before you ship.

Measuring too early. Checking after one week, before the page was even recrawled, and declaring failure. Or success. Both are just noise.

And the bounce-back effect. We often pick pages to fix because they just had a bad month. But a page at an unusual low point tends to drift back toward its normal level on its own, fix or no fix. If you compare against that low point, almost any change looks like a win. Use a longer baseline, and lean on the control pages, which had their own ups and downs in the same period.

Wrap-up

Metric chosen in advance, window from the recrawl, and always a control group. The change log and re-measure sheet are linked in the description. Next episode: we put the whole series together and run a full analysis on a real site, start to finish. See you there.