How to Scrape Customer Reviews: Marketplaces, Maps and Review Sites With Proxies

Review scraping is the automated gathering of openly published buyer feedback from shops, map listings and rating platforms, followed by converting stars, dates and comment bodies into a table that analysts can query. The technical core is simple: find where the page loads its reviews from, walk through every page or cursor, store each review once, and spread the requests over a sensible pace and, when needed, several IP addresses.

The legal and ethical side deserves equal attention. Reviews are written by real people, platforms set their own rules for automated access, and some of them sell this data through official channels. Below you will find where reviews live and how each source delivers them, a data model that keeps personal data to a minimum, pagination code, incremental updates, proxy settings and a first pass at sentiment analysis.

Ground Rules Before You Collect Anything

Treat these as part of the project scope, not an afterthought:

• Public reviews only. Stick to content an anonymous visitor is shown. Signing in to reach extra data, or getting around paywalls, access controls or CAPTCHAs, is out of bounds.
• Read the terms and robots.txt. Many platforms restrict automated collection in their terms of service. Where an official API or data licence exists, it is usually the safer route. When in doubt, ask a lawyer about your jurisdiction and use case.
• Personal data caution. Author names, profile URLs and avatars may count as personal information under European, Russian, Californian and comparable privacy rules. Store only the fields you need, pseudonymise author identifiers and set a retention period.
• Read, never write. A review scraper only reads. Posting, editing, voting on or generating reviews is outside the scope of this guide, and fake reviews are illegal in many countries.
• Respect capacity. Keep request rates low and stop when a site signals that you are too fast.

Where Reviews Live and How They Load

Exact limits of each official API change over time, so check the current documentation of the platform before you plan around them. If an API gives you everything you need for your own products, use it and skip scraping entirely.

What to Store: a Lean Review Record

Decide on the schema before writing code. A compact record keeps the dataset useful and limits how much personal data you hold:

Skip avatars, profile links and any contact details. If the text itself contains phone numbers or e-mail addresses, mask them during cleaning.

Step 1. Find the Real Source of the Reviews

Open a product or place page, press F12, switch to the Network tab and filter by Fetch/XHR. Now click the next page of reviews or scroll the review panel. A new request usually appears that returns JSON with ratings and texts. Note three things:

1. the URL and which parameter changes between pages (page, offset, cursor, pageToken);
2. the headers the request needs, for example a language header or a referer;
3. where the response tells you that there is nothing left (an empty list, a missing cursor, a hasMore flag).

When the network log shows nothing like that, the comments ship with the initial HTML, occasionally as a JSON blob tucked into a script element. Parsing that JSON is far more stable than chasing CSS classes, which change with every redesign.

Step 2. Walk Through Pages or Cursors

Most review feeds use one of two pagination styles. Numbered pages are easy: increase the counter until the list comes back empty. With cursors, each response carries an opaque marker for the following batch, and the loop echoes it back until the marker goes missing. The sketch below targets a placeholder endpoint in cursor mode; rename the fields to match the payload you inspected.

import os, time, random, hashlib
import requests

GATEWAY = os.environ["GATEWAY_URL"]          # http://user:pass@host:port
SALT = os.environ["AUTHOR_SALT"]
FEED = "https://reviews.example.com/v1/items/{item}/reviews"

session = requests.Session()
session.proxies = {"http": GATEWAY, "https": GATEWAY}
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
    "Accept": "application/json",
    "Accept-Language": "en-US,en;q=0.8",
})

def mask_author(raw_name: str) -> str:
    return hashlib.sha256((SALT + raw_name).encode()).hexdigest()[:16]

def pull_reviews(item_id: str, newest_known: str | None = None):
    token, batch_no = None, 0
    while True:
        params = {"sort": "newest", "limit": 20}
        if token:
            params["cursor"] = token
        reply = session.get(FEED.format(item=item_id), params=params, timeout=30)
        if reply.status_code == 429:
            time.sleep(int(reply.headers.get("Retry-After", 60)))
            continue
        reply.raise_for_status()
        payload = reply.json()
        for entry in payload.get("reviews", []):
            if entry["id"] == newest_known:
                return
            yield {
                "review_ref": entry["id"],
                "rating": entry.get("stars"),
                "posted_at": entry.get("date"),
                "body": entry.get("text", "").strip(),
                "author_key": mask_author(entry.get("author", "")),
            }
        token = payload.get("nextCursor")
        batch_no += 1
        if not token:
            return
        time.sleep(random.uniform(1.5, 4.0))

Pay attention to two choices here. Ordering by date together with the newest_known comparison makes repeat runs incremental, because collection halts at the first already stored item. The 429 handler, in turn, sleeps for the period the server requested rather than hammering it again at once.

Step 3. Store, De-duplicate and Update

Give the storage table a composite unique constraint over source and review_ref. Where a platform hides review identifiers, hash the rating, date and first 200 characters of text and use that as the key. Reviews can be edited or removed, so for long-running monitoring keep a last_seen date and periodically re-read the most recent pages rather than the whole history. For exports, JSON Lines keeps nested data intact and CSV opens directly in spreadsheets.

Step 4. Proxies: When You Need Them and Which Kind

A few hundred reviews for one product rarely need anything beyond polite pacing. Proxies become relevant in three situations:

• Volume. Monitoring thousands of products or places means many requests per day, and per-IP limits kick in.
• Geography. Map listings and marketplaces often show different reviews, languages or sort orders depending on the visitor's country. An exit IP in the target market gives you the view local customers have.
• Stability. Datacenter ranges are flagged quickly by large consumer platforms, so requests from them fail more often.

For review feeds, residential proxies with country targeting are the usual choice. Keep one sticky session for the whole cursor chain of a single item, because some feeds bind the cursor to the session, and rotate between items. Our article on sticky vs rotating proxies explains the trade-off, and the guide to proxies in web scraping covers pacing and headers in more depth. For Google properties specifically, see our page on proxies for Google.

Step 5. From Text to Insight: Sentiment and Topics

Star ratings alone hide most of the signal. Three steps turn raw comments into something a product or marketing team can act on:

1. Language detection and cleaning. Separate languages before analysis, remove markup and mask personal details.
2. Sentiment scoring. For English, the VADER lexicon in the vaderSentiment package gives a quick compound score from −1 to 1. Multilingual transformer models are more accurate for other languages but need more compute.
3. Aspect grouping. Count how often phrases about delivery, size, battery, staff or price appear in negative versus positive reviews. That shows what drives low ratings.

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

scorer = SentimentIntensityAnalyzer()

def tone(text: str) -> str:
    value = scorer.polarity_scores(text)["compound"]
    if value >= 0.05:
        return "positive"
    if value <= -0.05:
        return "negative"
    return "neutral"

Combined with prices and stock data, review trends are a staple of competitor research; the article on proxies in market analytics shows how teams set up such monitoring.

Common Problems