Category:Scraping Use Cases

Scrape Google Search Results Using Python (Organic, Paid, FAQs, Related Terms)

15 Mins Read

Created Date: June 18, 2025

Updated Date: March 26, 2026

Selman Gökçe

Head of Marketing

Every SEO team, ad analyst, and data pipeline eventually comes back to the same source.

Google Search is where your users go, where your competitors live, and where the most valuable data on the internet hides in plain sight.

But Google does not hand that data over easily.

From organic links and paid ads to FAQs and related terms, every SERP is packed with insights, if you know how to extract them.

The target data includes organic listings, ads, People Also Ask blocks, and related search terms. Two extraction approaches: raw HTML parsing with BeautifulSoup for full control, and the Scrape.do Google Search API for pre-parsed structured JSON.

All requests go through Scrape.do.

Find all code used in our GitHub repository.

Why You Need to Scrape Google Search Results

Search results aren’t static answers. They’re live, structured snapshots of what people care about right now.

Scraping them gives you access to that attention.

Improving SEO Strategy

If you're serious about search, relying on third-party rank trackers isn't enough.

Scraping actual SERPs lets you see how your pages are displayed, what titles and descriptions Google shows, which competitors appear next to you, and whether features like snippets or FAQs are pushing your links down.

You get full control and visibility on your terms, and in real time.

Reviewing Marketing Campaigns

Organic and paid listings live on the same page and if you’re running ads, you need to see how they show up.

Scraping lets you track your ad placements, identify what competitors are doing, monitor changes over time, and even archive creative variations. For e-commerce teams, Google Shopping scraping adds competitor pricing and seller data to the mix.

Whether you're running hundreds of campaigns or just a few key terms, it’s a vital layer of insight most dashboards don’t give you.

Training LLMs or AI Systems

For LLM pipelines or real-time generative tools, scraped SERPs are an excellent source of fresh, intent-rich text. Google's AI Overviews add another layer of structured data worth extracting.

You can use them to generate domain-specific corpora, build RAG agents that ground outputs with current info, or train ranking models on how Google orders results. For academic and research-heavy domains, Google Scholar results provide peer-reviewed data that strengthens model accuracy.

Search engines are optimized for relevance and scraping them means you’re capturing high-quality data curated by design.

Discovering Content and Trends

Google is a content aggregator.

If a new article drops, a product launches, or a trend starts moving, it will show up there first — often before it gets indexed elsewhere. Google News scraping is especially useful for tracking breaking stories and brand mentions across publishers.

Scraping lets you discover URLs you wouldn’t catch through crawling, and surface content aligned with real user searches. Google Trends data is especially useful for identifying rising search topics before competitors notice them.

Prerequisites & Setup

Before we start scraping, let’s make sure your environment is ready.

We’ll be using Python, along with two libraries:

requests to send HTTP requests
BeautifulSoup from bs4 to parse the HTML

Install them with:

pip install requests beautifulsoup4

We’re also going to use Scrape.do to handle Google’s anti-bot protections. Scrape.do takes care of:

rotating proxies,
header management,
and bypassing rate limits or redirects

...so you can send a single request and get a clean response.

If you don’t already have an API token, sign up here for free and get 1000 monthly credits. You’ll need to replace "<your_token>" in the code examples with your actual Scrape.do token.

Once you have your token and libraries installed, you’re ready to start.

Scraping First Page Organic Results

Let’s start with the core of Google Search: the organic results.

Each organic search result is rendered inside a div with the class Ww4FFb, which acts as a reliable anchor when parsing the HTML. Within this block, you’ll find the title, URL, and description; everything you need to build your own search scraper.

Here’s a simple script that does exactly that using Scrape.do and BeautifulSoup:

import requests
import urllib.parse
from bs4 import BeautifulSoup

# Your Scrape.do token and search query
scrape_token = "<your_token>"
query = "python web scraping"

# Encode the search query and build Google URL
encoded_query = urllib.parse.quote_plus(query)
google_url = f"https://www.google.com/search?q={encoded_query}&start=0"  # start=0 for first page

# Scrape.do wrapper URL - properly encode the Google URL
api_url = f"https://api.scrape.do/?token={scrape_token}&url={urllib.parse.quote(google_url, safe='')}"

# Send the request
response = requests.get(api_url)
response.raise_for_status()

# Parse the HTML
soup = BeautifulSoup(response.text, 'html.parser')

# Find all search results with Ww4FFb class
search_results = soup.find_all('div', class_=lambda x: x and 'Ww4FFb' in x)

# Extract data from each result
for position, result in enumerate(search_results, 1):
    # Get title from h3 tag
    title = result.find('h3').get_text(strip=True)

    # Get URL from link
    url = result.find('a').get('href')

    # Get description/snippet
    desc_element = result.find(class_='VwiC3b')
    description = desc_element.get_text(strip=True) if desc_element else "No description"

    print(f"{position}. {title}")
    print(f"   URL: {url}")
    print(f"   Description: {description}")
    print()

When run, this will print something like:

1. Python Web Scraping Tutorial
   URL: https://www.geeksforgeeks.org/python-web-scraping-tutorial/
   Description: Jan 2, 2025—Python web scraping refers to the process of extracting data from websites using Python...

2. How to start Web scraping with python? : r/learnpython
   URL: https://www.reddit.com/r/learnpython/comments/qzr8ir/how_to_start_web_scraping_with_python/
   Description: Learn the basic html elements that build up a website. Inspect the element on the webpage that...

3. Python Web Scraping: Full Tutorial With Examples (2025)
   URL: https://www.scrapingbee.com/blog/web-scraping-101-with-python/
   Description: May 27, 2025—Learn about web scraping in Python with this step-by-step tutorial...

...

...

You now have a working scraper that extracts the full set of organic results from the first page of Google. In the next step, we’ll expand it to go beyond page one.

Scraping All Organic Google Search Results

Scraping a single page is useful but often not enough.

If you want the full picture, you'll need to go beyond the first 10 results and paginate through all available pages.

Fortunately, Google structures its result pages in a predictable way: each new page is accessible by increasing the start parameter by 10 (e.g., start=10, start=20, etc.).

We’ll take the code from the previous step and modify it so it loops through each page until there are no more results.

Here’s what we changed:

Introduced a while True loop to go through result pages
Automatically stops when no new results are found
Stores and prints every result with position, title, URL, and description

Here’s the complete code:

import requests
import urllib.parse
from bs4 import BeautifulSoup
import time

# Your Scrape.do token and search query
scrape_token = "<your_token>"
query = "python web scraping"

# Starting page
start = 0
all_results = []

while True:
    print(f"Scraping page {(start // 10) + 1} (results {start + 1}-{start + 10})...")

    # Encode the search query and build Google URL
    encoded_query = urllib.parse.quote_plus(query)
    google_url = f"https://www.google.com/search?q={encoded_query}&start={start}"

    # Scrape.do wrapper URL - properly encode the Google URL
    api_url = f"https://api.scrape.do/?token={scrape_token}&url={urllib.parse.quote(google_url, safe='')}"

    # Send the request
    response = requests.get(api_url)
    response.raise_for_status()

    # Parse the HTML
    soup = BeautifulSoup(response.text, 'html.parser')

    # Find all search results with Ww4FFb class
    search_results = soup.find_all('div', class_=lambda x: x and 'Ww4FFb' in x)

    # Break if no results found
    if not search_results:
        print("No more results found!")
        break

    # Extract data from each result
    for result in search_results:
        title = result.find('h3').get_text(strip=True)
        url = result.find('a').get('href')
        desc_element = result.find(class_='VwiC3b')
        description = desc_element.get_text(strip=True) if desc_element else "No description"

        all_results.append({
            'position': len(all_results) + 1,
            'title': title,
            'url': url,
            'description': description
        })

    start += 10
    time.sleep(1)  # optional delay to mimic human browsing

# Print all results
print(f"\n=== Found {len(all_results)} total results ===\n")

for result in all_results:
    print(f"{result['position']}. {result['title']}")
    print(f"   URL: {result['url']}")
    print(f"   Description: {result['description']}")
    print()

This script goes through each search result page and stops automatically when it reaches the end.

The next target is paid ads. Scraping those requires a slightly different approach.

Scraping Search Ads from Google Search

Paid results on Google (search ads) appear above or below the organic results. They’re often just as valuable, especially if you're analyzing competitor campaigns, tracking your own ads, or building datasets for ad intelligence.

Unlike organic results, ads don’t follow the same HTML structure. They live inside elements with the class uEierd, and the internal layout can vary slightly between different ad formats.

But there’s a pattern.

We’ll use Scrape.do to fetch the page, then extract all ad containers, and inside each, we’ll look for:

The ad’s destination URL
Its title
The description text
The displayed (vanity) URL

Let’s walk through the code:

import requests
import urllib.parse
from bs4 import BeautifulSoup

# Your Scrape.do token and search query
scrape_token = "<your_token>"
query = "python web scraping"

# Encode the search query and build Google URL
encoded_query = urllib.parse.quote_plus(query)
google_url = f"https://www.google.com/search?q={encoded_query}"

# Scrape.do wrapper URL
api_url = f"https://api.scrape.do/?token={scrape_token}&url={urllib.parse.quote(google_url, safe='')}"

# Send the request
response = requests.get(api_url)
response.raise_for_status()

# Parse the HTML
soup = BeautifulSoup(response.text, 'html.parser')

# Find all search ads using uEierd class
search_ads = soup.find_all('div', class_='uEierd')

print(f"Found {len(search_ads)} search ads")

# Extract ad information
for position, ad in enumerate(search_ads, 1):
    print(f"\n--- Ad {position} ---")

    # Extract URL
    url_element = ad.find('a')
    url = url_element.get('href') if url_element else "URL: Not found"
    print(f"URL: {url}")

    # Extract title
    title_element = ad.find(['h3', 'div'], class_=lambda x: x and any(cls in str(x) for cls in ['CCgQ5', 'vCa9Yd', 'QfkTvb']))
    if not title_element:
        title_element = ad.find(['h1', 'h2', 'h3', 'h4', 'h5', 'h6'])
    title = title_element.get_text(strip=True) if title_element else "Title: Not found"
    print(f"Title: {title}")

    # Extract description
    description_element = ad.find('div', class_=lambda x: x and any(cls in str(x) for cls in ['Va3FIb', 'r025kc', 'lVm3ye']))
    if not description_element:
        description_spans = ad.find_all('span')
        description_texts = [
            span.get_text(strip=True)
            for span in description_spans
            if len(span.get_text(strip=True)) > 20 and span.get_text(strip=True) not in ['Sponsored', 'Ad']
        ]
        description = ' '.join(description_texts) if description_texts else "Description: Not found"
    else:
        description = description_element.get_text(strip=True)
    print(f"Description: {description}")

    # Extract displayed (vanity) URL
    display_url_element = ad.find('span', class_=lambda x: x and 'qzEoUe' in str(x))
    display_url = display_url_element.get_text(strip=True) if display_url_element else "Display URL: Not found"
    print(f"Display URL: {display_url}")

And here’s a simulated output for two sample ads (for clarity):

--- Ad 1 ---
URL: https://www.datacamp.com/learn/python
Title: Learn Python for Data Science | Online Courses - DataCamp
Description: Gain hands-on experience with Python. Start learning with interactive exercises and projects today.
Display URL: www.datacamp.com

--- Ad 2 ---
URL: https://www.udemy.com/course/python-for-data-science/
Title: Python for Data Science | Udemy
Description: Enroll in our bestselling course. Taught by industry experts. Lifetime access included.
Display URL: www.udemy.com

These are examples. The structure stays the same across most queries.

The remaining data types are Google’s FAQ blocks and related search terms: two highly underrated sources of insight.

Google doesn’t just show links but it shows what users are really asking.

Boxes like “People also ask” and related searches at the bottom of the page give you direct insight into search intent.

They reveal follow-up questions, keyword variations, and what else your audience is curious about. For SEO, content strategy, and user research, this data is gold.

Scrape Frequently Asked Questions from Google

Google’s FAQ block, the one with expandable questions, uses a specific JS tag called yEVEwb.

This is a good target for extracting the question texts themselves. While answers are loaded dynamically when you expand them, the questions are already embedded in the page, so we can scrape them without JavaScript rendering.

Here’s the code:

import requests
import urllib.parse
from bs4 import BeautifulSoup

# Your Scrape.do token and search query
scrape_token = "<your_token>"
query = "python web scraping"

# Encode the search query and build Google URL
encoded_query = urllib.parse.quote_plus(query)
google_url = f"https://www.google.com/search?q={encoded_query}"

# Scrape.do wrapper URL - properly encode the Google URL
api_url = f"https://api.scrape.do/?token={scrape_token}&url={urllib.parse.quote(google_url, safe='')}"

# Send the request
response = requests.get(api_url)
response.raise_for_status()

# Parse the HTML
soup = BeautifulSoup(response.text, 'html.parser')

# Find all FAQ sections with yEVEwb
faq_results = soup.find_all('div', jsname='yEVEwb')

# Extract FAQ questions
for position, faq in enumerate(faq_results, 1):
    question_element = faq.find('span')
    if question_element:
        question = question_element.get_text(strip=True)
        print(f"{position}. {question}")

Here’s what the output looks like:

1. Is Python good for web scraping?
2. How to scrape the web with Python?
3. Why is data scraping illegal?
4. How long does it take to learn web scraping in Python?

If you scroll to the bottom of any Google results page, you’ll see a block of suggested queries, often labeled “Searches related to...”

These related terms are excellent for expanding your keyword coverage. They reflect how users explore a topic, what phrases they use, and where their curiosity goes next.

To extract them, we look for div elements with the classes b2Rnsc vIifob. Each one contains a single search suggestion.

Here’s how to scrape them:

import requests
import urllib.parse
from bs4 import BeautifulSoup

# Your Scrape.do token and search query
scrape_token = "<your_token>"
query = "python web scraping"

# Encode the search query and build Google URL
encoded_query = urllib.parse.quote_plus(query)
google_url = f"https://www.google.com/search?q={encoded_query}"

# Scrape.do wrapper URL
api_url = f"https://api.scrape.do/?token={scrape_token}&url={urllib.parse.quote(google_url, safe='')}"

# Send the request
response = requests.get(api_url)
response.raise_for_status()

# Parse the HTML
soup = BeautifulSoup(response.text, 'html.parser')

# Find all related search terms
related_searches = soup.find_all('div', class_='b2Rnsc vIifob')

print(f"Found {len(related_searches)} related search terms")

# Extract related search terms
for position, search_term in enumerate(related_searches, 1):
    term_text = search_term.get_text(strip=True)
    print(f"{position}. {term_text}")

And here’s the sample output:

Found 8 related search terms

1. Python web scrapingbook
2. Python web scrapinglibrary
3. Python web scrapingSelenium
4. Web scrapingusingPythonBeautifulSoup
5. Web scraping Pythonw3schools
6. Python web scrapinggithub
7. Web scrapingwithPythonPDF
8. Python web scrapingexamples

You can use these suggestions to generate long-tail keywords, cluster related content, or explore adjacent topics your users are already searching for.

Extract All SERP Data with the Google Search API

Here is the thing about the scrapers above: they work today. But Google rotates CSS class names without warning, and DOM mutations happen without changelog entries. Every selector is a maintenance liability.

For production pipelines where uptime matters more than custom extraction logic, a structured SERP API eliminates this overhead. The Scrape.do Google Search API returns all standard SERP result types as structured JSON from a single endpoint. One request replaces five separate scrapers.

curl "https://api.scrape.do/plugin/google/search?token=<your_token>&q=scrape+google+search+results&gl=us&hl=en"

The response is a JSON object containing every element Google rendered for that query:

{
  "search_parameters": { "query, language, country, device, pagination offset" },
  "search_information": { "page title, total results, time taken, country, city" },
  "ai_overview": {
    "state": "complete or deferred",
    "text_blocks": [
      {
        "type": "paragraph or list",
        "snippet": "AI-generated summary text for this block",
        "reference_indexes": ["indexes pointing to the references array"],
        "list": ["nested list items when type is list"]
      }
    ],
    "references": [
      { "title": "source page title", "link": "source URL", "snippet": "excerpt", "source": "domain" }
    ]
  },
  "organic_results": [
    { "position": 1, "title": "page title", "link": "URL", "snippet": "description text", "displayed_link": "visible URL", "sitelinks": ["sub-page links when present"] }
  ],
  "top_ads": [
    { "position": 1, "title": "ad headline", "link": "landing page URL", "description": "ad copy", "displayed_link": "visible URL", "tracking_link": "click tracking URL" }
  ],
  "bottom_ads": [
    { "position": 1, "title": "ad headline", "link": "landing page URL", "description": "ad copy" }
  ],
  "related_questions": [
    { "question": "People Also Ask question text", "snippet": "expanded answer", "link": "source URL", "title": "source page title" }
  ],
  "related_searches": [
    { "query": "related search term", "link": "Google search URL for this term" }
  ],
  "knowledge_graph": { "title, type, description, source, image, attributes" },
  "local_map": { "link, image, gps_coordinates" },
  "local_results": [
    { "title": "business name", "address": "full address", "phone": "number", "rating": 4.5, "reviews": 120, "type": "business category" }
  ],
  "top_stories": [
    { "title": "headline", "link": "article URL", "source": "publisher", "date": "publish date", "thumbnail": "image URL" }
  ],
  "video_results": [
    { "title": "video title", "link": "URL", "source": "platform", "thumbnail": "image URL", "duration": "length" }
  ],
  "discussions_and_forums": [
    { "title": "thread title", "link": "URL", "source": "platform", "snippet": "preview text" }
  ],
  "pagination": { "current page, next page URL, other page URLs" },
  "navigation": [
    { "title": "tab name (All, Images, News, Maps, Videos...)", "link": "URL" }
  ]
}

Every field the five HTML scrapers extract individually (organic_results, top_ads, bottom_ads, related_questions, related_searches) is present in this single response, along with local business data from Google Maps and publisher headlines. The ai_overview field is null when Google does not generate one for the query. When it exists, the state is either complete (full content inline) or deferred (Google is still generating it, and a follow-up request to /plugin/google/search/ai-overview with the returned session_key fetches the completed content for 5 additional credits; the key expires after 60 seconds and is single-use).

The raw HTML approach from earlier sections gives full control over extraction: you can target non-standard elements, capture layout-specific data, and parse anything visible on the page. The API approach gives zero selector maintenance, consistent field names, and a single request for all data types. If you're evaluating other providers, the SerpApi alternatives comparison covers pricing, output richness, and feature gaps across eight services. Use whichever fits the pipeline.

FAQ

Is scraping Google Search results legal?

Scraping publicly available search results is generally permissible under the US Computer Fraud and Abuse Act, as confirmed by the hiQ Labs v. LinkedIn ruling. Google’s Terms of Service prohibit automated access, but courts have consistently held that public data scraping does not constitute unauthorized access. Always consult legal counsel for your specific jurisdiction and use case.

Why does Google return different results for the same query?

Google personalizes results based on location, search history, device type, and language settings. Two requests from different IP addresses or with different gl (country) and hl (language) parameters will return different SERPs. Using Scrape.do’s geoCode parameter or the API’s gl and hl fields gives you control over which localized results you receive.

What happens when Google changes its HTML structure?

Google rotates CSS class names and DOM structures without notice. A selector that works today may break tomorrow. This is the primary argument for using the Google Search API: it abstracts away HTML changes and returns consistent JSON field names regardless of what Google changes on the frontend.

How do I handle rate limits and 429 errors?

Google rate-limits aggressive request patterns. Scrape.do handles proxy rotation and request pacing internally, but adding a 1-2 second delay between paginated requests reduces the chance of triggering additional bot detection. The super=true parameter activates premium residential proxies for queries that hit stricter rate limits.

Can I scrape Google Shopping, Maps, or News with the same approach?

The HTML parsing approach works across Google properties, but each uses different DOM structures and selectors. Maps uses protobuf endpoints, Shopping relies on async AJAX pagination, and News has its own container structure. Each vertical is covered in its own guide earlier in this article. The SERP API covers multiple Google properties from a single endpoint.

Conclusion

Scraping Google Search gives you unmatched access to what people are searching, seeing, and clicking.

From organic results to ads, FAQs, and related terms, two approaches cover the full extraction spectrum. Raw HTML parsing with BeautifulSoup gives full control over every element on the page. The Google Search API returns all standard SERP types as structured JSON from a single endpoint, with zero selector maintenance.

And with Scrape.do, you don’t need to worry about: