Data wrangling is the work between raw data and analysis: gathering, structuring, cleaning, normalising, filtering, enriching and validating. In Python it is done with pandas. Below, every step runs on 62 real listings scraped on 20 November 2026 from two marketplaces, in two languages and two currencies, ending with 45 comparable rows.
Teaching examples start from a tidy CSV. Real data does not: titles carry emoji and German, prices come in euros and dollars, a quarter of the rows are off target, and a few are duplicates. This post shows the script we ran and its output, so each step has a visible reason.
Data wrangling steps, and how they differ from cleaning
Cleaning is one step of wrangling. Wrangling is the whole route from "rows from several places" to "one table I trust".
| Step | Question it answers | In this example |
|---|---|---|
| Gather | Where does each row come from? | Three collector runs: eBay.de, Amazon.com, the ECB rate |
| Structure | Do all rows share one schema? | Same columns for both marketplaces, plus a source column |
| Clean | Is the text usable? | Emoji and symbols out of the titles, spaces collapsed |
| Normalise | Are values comparable? | Every price in US dollars |
| Filter | Is every row on target? | Accessories, games and other models removed |
| Enrich | What can I derive? | Condition and bundle flags from title words in English and German |
| Validate | Are there duplicates or impossible values? | Exact duplicates dropped, outliers cut per source |
"Normalisation" means two things in the search results. In databases it is splitting tables into normal forms. In analysis, and here, it is putting values on one scale or unit, such as one currency, so they can be compared.
Gather: three collector runs
We searched both marketplaces for the same product, a Nintendo Switch OLED console, with the eBay collector on the German site and the Amazon collector on the US site, and fetched the day's euro to dollar reference rate with the exchange rates collector. All three ran from Claude through our MCP server:
run_collector { "slug": "ebay_search", "input": { "query": "nintendo switch oled", "country": "de" } }
run_collector { "slug": "amazon_search", "input": { "query": "nintendo switch oled", "country": "us" } }
run_collector { "slug": "exchange_rates", "input": { "base": "EUR", "symbols": ["USD"], "max_results": 1 } }
They returned 40 eBay.de listings, 22 Amazon.com products and one rate row (EUR to USD 1.099, dated 20 November 2026). At the published prices that is $0.12, $0.022 and $0.0002. A raw eBay.de title looks like Nintendo Switch OLED | V1/V2 | verschiedene Farben | Auswahl | TOP Zustand!, with a game-controller emoji after "V1/V2" and a check-mark emoji at the end.
The script
The script reads the three runs by id over the REST API and applies the steps in order. Note the cleaning line. With pandas 3.0.3, the version we ran, strings are backed by PyArrow, and in our first attempt a pandas str.replace with \w stripped the German umlauts ("Weiß" became "Wei"). Python's re keeps them, so the title cleaner uses it.
import os, re, sys, requests, pandas as pd
sys.stdout.reconfigure(encoding="utf-8")
API = "https://api.quanticdata.io/v1/scraper/collectors/runs/"
headers = {"Authorization": f"Bearer {os.environ['QUANTICDATA_API_KEY']}"}
def rows(run_id):
return requests.get(API + run_id, headers=headers, timeout=60).json()["payload"]["results"]
# 1. Gather: two marketplaces, two shapes, two currencies (runs of 20 Nov 2026)
ebay_de = pd.DataFrame(rows("EBAY_RUN_ID")) # ebay_search, country de
amazon_us = pd.DataFrame(rows("AMAZON_RUN_ID")) # amazon_search, country us
eur_usd = rows("FX_RUN_ID")[0]["rate"] # exchange_rates, ECB
print("raw rows:", len(ebay_de), "eBay.de +", len(amazon_us), "Amazon.com")
# 2. Structure: one schema for both sources
cols = ["title", "price_value", "currency"]
df = pd.concat([ebay_de[cols].assign(source="ebay.de"),
amazon_us[cols].assign(source="amazon.com")], ignore_index=True)
# 3. Clean: strip emoji and symbols, collapse spaces, lowercase copy for matching
clean = lambda s: re.sub(r"\s+", " ", re.sub(r"[^\w\s/+&.,()\-\"']", " ", s)).strip()
df["title"] = df["title"].map(clean) # Python re: \w keeps umlauts and accents
t = df["title"].str.lower()
# 4. Normalize: one currency
df["price_usd"] = df["price_value"].where(df["currency"] == "USD",
df["price_value"] * eur_usd).round(2)
# 5. Filter: OLED consoles only, no Switch 2, no accessories or games
accessory = r"protector|case for|silicone|accessories kit|charger|exclusive edition"
keep = t.str.contains("oled") & ~t.str.contains("switch 2") & ~t.str.contains(accessory)
print("dropped as off-target:", int((~keep).sum()))
df, t = df[keep].copy(), t[keep]
# 6. Enrich: condition and bundle flags from the title text (EN + DE words)
used = r"renewed|refurbished|used|zustand|b-ware|gebraucht|genutzt|getestet"
df["condition"] = t.str.contains(used).map({True: "used/renewed", False: "not stated"})
df["bundle"] = t.str.contains(r"\+|\bspiele?\b|mario kart|inkl|set mit|zubeh")
# 7. Deduplicate and drop price outliers (IQR rule per source)
before = len(df)
df = df.drop_duplicates(subset=["source", "title", "price_usd"])
q = df.groupby("source")["price_usd"].quantile([.25, .75]).unstack()
lo = df["source"].map(q[0.25] - 1.5 * (q[0.75] - q[0.25]))
hi = df["source"].map(q[0.75] + 1.5 * (q[0.75] - q[0.25]))
df = df[df["price_usd"].between(lo, hi)]
print("duplicates and outliers removed:", before - len(df))
print("clean rows:", len(df), "\n")
summary = (df.groupby(["source", "condition"])["price_usd"]
.agg(listings="count", median="median", min="min", max="max").round(2))
print(summary.to_string())
print("\nbundles (console + game or accessories):", int(df["bundle"].sum()), "of", len(df))
df.to_csv("switch_oled_clean.csv", index=False)
The output
raw rows: 40 eBay.de + 22 Amazon.com
dropped as off-target: 16
duplicates and outliers removed: 1
clean rows: 45
listings median min max
source condition
amazon.com not stated 3 387.99 341.50 414.99
used/renewed 4 277.99 268.95 286.00
ebay.de not stated 29 203.32 158.74 295.63
used/renewed 9 204.40 124.13 251.67
bundles (console + game or accessories): 16 of 45
And three of the 45 clean rows, as the CSV holds them:
| source | title (cleaned) | price | price_usd | condition | bundle |
|---|---|---|---|---|---|
| ebay.de | Nintendo Switch OLED Konsole Weiss komplett + 1x Spiel Auswahl (Zelda, Mario..) | 149.00 EUR | 163.75 | not stated | True |
| ebay.de | Nintendo Switch OLED V1/V2 verschiedene Farben Auswahl Zustand "gut" | 112.95 EUR | 124.13 | used/renewed | False |
| amazon.com | Nintendo Switch OLED Model w/ White Joy-Con (Renewed) | 279.99 USD | 279.99 | used/renewed | False |
What each step changed
- Filtering mattered most. A marketplace search returns what sells next to the product. 16 of 62 rows were screen protectors, cases, chargers, games, the newer Switch 2 or older models.
- Normalising the currency made the two sources comparable at all. Note what it does not fix: the two marketplaces differ in sellers, taxes and shipping, so a lower median on one site is not by itself a cheaper market. Normalising a unit is not the same as normalising a market.
- Enrichment from text in two languages flagged 13 used or renewed listings from title words, using "Zustand" (condition) and "B-Ware" on the German side and "Renewed" on the US side, and 16 bundles with a game or accessories.
- Validation removed one row: a duplicate or an outlier beyond 1.5 times the interquartile range of its marketplace. Small, but on a daily price feed duplicates add up.
Data wrangling tools, and when to change them
| Tool | Use it when |
|---|---|
| pandas | Up to a few million rows in memory, the default for scraped data |
| SQL in your warehouse | The rows already live there and the logic is joins and filters |
| Spreadsheet | A one-off check by someone who does not code |
| Collector output fields | The source already gives typed values, such as a numeric price_value and a currency, so parsing is done before wrangling starts |
The last row is why we started from collector output instead of raw HTML: prices arrived as numbers with an ISO currency code, which removed a whole parsing step. If you start from raw pages, see our guide to data parsing first. To run the same steps every day, wrap them in a scheduled job as described in how data pipelines work, and for the business side, see competitor price monitoring and how to build a price monitor. The same cleaning applies to text data, as in sentiment analysis of customer reviews.


