Live SERP API: Get Real Results Programmatically

Full transcript

1,016 words · script draft, under review.

Intro

Last episode we analysed ten results pages by hand. It took an hour, and it was worth it.

Now you have four hundred keywords. By hand, that's a week. And next month you want to do it again, to see what changed.

You also can't save a results page from a browser in any useful way. A screenshot can't be sorted, filtered or compared.

In this episode we'll pull live Google results through the DataForSEO API: which parameters matter, what it costs, how to read the response by item type, and how to store each results page as a snapshot you can compare later.

The endpoint and its parameters

We set up the account and made a first call in episode three. Now we use one specific endpoint: SERP, Google, organic, live, advanced. "Live" means you get the answer in the same request. "Advanced" means you get every element on the page, not just the blue links.

The request is a list of tasks. Each task describes one search, with a handful of parameters that matter.

The keyword, exactly as a person would type it.

The location. You can give it as a name or a numeric code. The code for the United States is twenty-eight forty. Results differ a lot between countries, and for local topics between cities, so always set this explicitly.

The language code, for example "en".

The device: desktop or mobile. The results can be different, and we'll compare them properly in episode thirty-four.

And the depth: how many results you want. Ten gives you page one. Thirty gives you the first three pages.

What it costs

Depth is the main cost lever. In my own billing, a live request at depth ten costs about a third of a cent. At depth thirty, about twice that.

So four hundred keywords at depth ten is roughly a dollar twenty. At depth thirty, around two forty. Cheap, but it adds up if you rerun every week, or if you set the depth to a hundred without thinking.

My rule is simple. Use depth ten for analysis. Use thirty only when you specifically need to see pages two and three, like when you're tracking pages in striking distance from episode ten. And before every run, the script prints the number of keywords times the price and waits for a yes. That's the cost gate from episode six.

The response also tells you what each task actually cost. Log that number. It's the real price, and it tells you when the price list has changed.

Reading the response

The response is nested, but the part you need is simple. Inside tasks, then result, there's a list called items. Each item is one element on the results page, in the order it appears.

Every item has a type. "Organic" is a normal blue link. Others are things like "featured snippet", "people also ask", "local pack", "video", "images" and "AI overview". Episode twenty-eight goes through what each of them means.

Each item also has two positions, and mixing them up is the most common mistake. Rank group is the position within its own type. The third organic result has rank group three. Rank absolute is the position among all elements on the page. If a snippet, a video row and a question box come first, that same third organic result might have rank absolute six.

Neither matches Search Console's average position exactly. Search Console averages over many searches, devices and days. The API gives you one snapshot. Use the API to see what the page looks like, and Search Console to see what actually happened.

For organic items, you'll mostly use the domain, the URL, the title and the description. That's enough for clustering, for classifying who ranks, and for title match.

My parser does one simple thing first: it loops over the items and sorts them into buckets by type. Organic items go to one list, everything else to a features list, with the position kept. Never assume the first item is the first organic result. On many results pages it isn't, and that alone breaks a lot of home-made rank trackers.

The result also lists which item types appear on the page, before you even open the items. That's a quick way to flag keywords with a map, a video row or an AI summary without parsing anything.

Storing snapshots

Store every response twice.

First, the raw response as a file in your cache folder, named by keyword, location, device and date. That's your receipt. If you later want a field you didn't parse, you don't have to pay again.

Second, the parsed items as rows in a SQLite table. One row per item, with the keyword, location, device, date, type, both positions, domain, URL and title. One results page becomes one snapshot, ten to thirty rows plus the features.

Now the questions from the worksheet become simple queries. Who ranks most often across my keywords? Which keywords have a video row? Where does the first organic result start? And because every row has a date, next month's run can be compared with this month's, keyword by keyword.

One more habit. Check the status of every task, not just the whole request. A request can succeed while a single task fails, for example because of a misspelled location. Write failed tasks to a list and retry only those.

Pitfalls

To sum up the traps. Forgetting the location, so you analyse the wrong country. Using a high depth by default and paying for pages nobody reads. Mixing rank group and rank absolute in the same report. And comparing snapshots taken with different settings.

None of these show up as errors. They show up as conclusions that are slightly wrong. That's why the settings are stored on every row.

Wrap-up

So: set location, language, device and depth on purpose, gate the cost, and store every results page as a dated snapshot. The Python script for fetching, parsing and storing is linked in the description. Next episode: what all those other item types mean — snippets, question boxes, local packs and AI Overviews — and what they do to your clicks. See you there.