Panel data is a dataset that observes the same entities, such as people, firms, products or countries, at several points in time. Each row is one entity in one period, so it combines a cross-section with a time series. A real example: GDP per capita and inflation for five countries over nine years.
That example is the one built below. On 20 November 2026 we pulled two World Bank indicators for Germany, Spain, France, Italy and Portugal from 2015 to 2023 with two collector runs, then built and analysed the panel in pandas. All the tables are the script's real output.
Panel data vs cross-sectional data vs time series
| Type | Entities | Periods | Example from this post |
|---|---|---|---|
| Cross-sectional | Many | One | Inflation in all five countries in 2022 |
| Time series | One | Many | Inflation in Italy from 2015 to 2023 |
| Pooled cross-section | Many, different each period | Many | A survey of new households every year |
| Panel | Many, the same each period | Many | Inflation in all five countries, every year from 2015 to 2023 |
The point of following the same entities is that you can separate what differs between them (Germany is richer than Portugal every year) from what changes within each of them over time (every country's inflation jumped in 2022). A cross-section cannot tell those apart, and a single time series has nothing to compare against.
Panels are balanced when every entity is observed in every period, and unbalanced when some entity-period pairs are missing. They are long when each row is one entity-period, and wide when each entity or each period becomes a column.
Building a panel dataset: two collector runs
The World Bank Indicators API serves each indicator by country and year. Our World Bank collector wraps it: one indicator code, a list of countries and a year range, one row per country-year. We ran it twice from Claude through the QuanticData MCP server:
run_collector { "slug": "world_bank", "input": { "indicator": "NY.GDP.PCAP.CD",
"countries": ["IT", "DE", "FR", "ES", "PT"], "year_from": 2015, "year_to": 2023, "max_results": 45 } }
run_collector { "slug": "world_bank", "input": { "indicator": "FP.CPI.TOTL.ZG",
"countries": ["IT", "DE", "FR", "ES", "PT"], "year_from": 2015, "year_to": 2023, "max_results": 45 } }
Each returned 45 datapoints. NY.GDP.PCAP.CD is GDP per capita in current US dollars; FP.CPI.TOTL.ZG is consumer price inflation in percent. One row as delivered:
{"country": "Germany", "country_code": "DEU", "country_iso2": "DE", "year": "2023",
"value": 54776.7668235491, "indicator": "NY.GDP.PCAP.CD",
"indicator_name": "GDP per capita (current US$)", "unit": null, "obs_status": null}
At $0.0002 per datapoint the 90 datapoints cost $0.018.
The Python script
import os, sys, requests, numpy as np, pandas as pd
sys.stdout.reconfigure(encoding="utf-8")
API = "https://api.quanticdata.io/v1/scraper/collectors/runs/"
headers = {"Authorization": f"Bearer {os.environ['QUANTICDATA_API_KEY']}"}
def rows(run_id):
return requests.get(API + run_id, headers=headers, timeout=60).json()["payload"]["results"]
# Two world_bank runs of 20 Nov 2026: IT, DE, FR, ES, PT, 2015-2023
gdp = pd.DataFrame(rows("GDP_RUN_ID")) # NY.GDP.PCAP.CD, GDP per capita, current US$
cpi = pd.DataFrame(rows("CPI_RUN_ID")) # FP.CPI.TOTL.ZG, inflation, consumer prices %
# 1. Long format: one row per entity (country) and period (year)
long = (gdp[["country_code", "year", "value"]].rename(columns={"value": "gdp_pc"})
.merge(cpi[["country_code", "year", "value"]].rename(columns={"value": "inflation"}),
on=["country_code", "year"]))
long["year"] = long["year"].astype(int)
panel = long.set_index(["country_code", "year"]).sort_index()
print(panel.shape[0], "observations:", panel.index.get_level_values(0).nunique(), "countries x",
panel.index.get_level_values(1).nunique(), "years")
counts = panel.groupby(level=0).size()
print("balanced panel:", counts.nunique() == 1, "\n")
print(panel.loc[["DEU", "ITA"]].loc[(slice(None), [2019, 2020, 2021, 2022, 2023]), :].round(1).to_string(), "\n")
# 2. Wide format: countries as columns, years as rows
print(panel["inflation"].unstack(level=0).round(1).to_string(), "\n")
# 3. Growth needs the previous year of the SAME country: shift within each entity
panel["gdp_growth"] = panel.groupby(level=0)["gdp_pc"].pct_change() * 100
# 4. Pooled vs within (fixed effects): demean each variable by country, then regress
d = panel.dropna()
pooled = np.polyfit(d["inflation"], d["gdp_growth"], 1)[0]
within = d - d.groupby(level=0).transform("mean")
fe = np.polyfit(within["inflation"], within["gdp_growth"], 1)[0]
print(f"slope of GDP-per-capita growth on inflation: pooled {pooled:.2f}, within countries {fe:.2f}")
print(f"variance of inflation that is within countries: {within['inflation'].var() / d['inflation'].var():.0%}")
# 5. Two-way: also remove what all five countries share in a given year
two_way = within - within.groupby(level=1).transform("mean")
tw = np.polyfit(two_way["inflation"], two_way["gdp_growth"], 1)[0]
print(f"slope with country and year effects removed: {tw:.2f}")
The output: long, wide and the balance check
45 observations: 5 countries x 9 years
balanced panel: True
gdp_pc inflation
country_code year
DEU 2019 47656.2 1.4
2020 47394.9 0.1
2021 52349.2 3.1
2022 50506.5 6.9
2023 54776.8 5.9
ITA 2019 33812.8 0.6
2020 32091.5 -0.1
2021 36852.5 1.9
2022 35653.9 8.2
2023 39279.7 5.6
country_code DEU ESP FRA ITA PRT
year
2015 0.5 -0.5 0.0 0.0 0.5
2016 0.5 -0.2 0.2 -0.1 0.6
2017 1.5 2.0 1.0 1.2 1.4
2018 1.7 1.7 1.9 1.1 1.0
2019 1.4 0.7 1.1 0.6 0.3
2020 0.1 -0.3 0.5 -0.1 -0.0
2021 3.1 3.1 1.6 1.9 1.3
2022 6.9 8.4 5.2 8.2 7.8
2023 5.9 3.5 4.9 5.6 4.3
slope of GDP-per-capita growth on inflation: pooled 0.17, within countries 0.17
variance of inflation that is within countries: 99%
slope with country and year effects removed: 0.64
The first block is the long form, indexed by entity and period, which is what panel models expect. The second is the wide form of one variable: read across a row for a cross-section, down a column for a time series. The panel is balanced: all five countries have all nine years.
What the panel shows that a cross-section cannot
- Almost all the variation in inflation is over time, not between countries. 99% of its variance is left after removing each country's average. In the wide table every column moves together: near zero in 2015 and 2020, then 5.2% to 8.4% in 2022.
- Shared shocks hide the relationship. The slope of GDP-per-capita growth on inflation is 0.17 both pooled and within countries, because both are dominated by the same year-to-year swings. Removing the year effects as well, so that each country is compared with the other four in the same year, gives 0.64.
- The unit matters. GDP per capita here is in current US dollars, so part of each country's yearly change is the euro to dollar exchange rate, which all five share. That is one reason the year effects are so large.
Five countries and nine years make a teaching example, not an estimate: the slopes come from 40 growth observations, and none of these numbers is a causal effect. The method is the same at any size.
Panel data pitfalls in pandas
- Lags and growth must stay inside each entity.
pct_change()on the whole column would compute Spain's 2015 growth from Germany's 2023 value, the row above it. Group by the entity level first, as in step 3. - Check balance before you model. A missing year shifts every lag after it. Count periods per entity, and decide whether to drop or fill.
- Keep the year an integer. The World Bank returns years as strings. String years sort correctly only until you mix formats; convert once, early.
- Write down the units. Current or constant prices, percent or index: the indicator name says, so keep it with the data.
The same structure works for business data: products observed every week with their price and rank, stores observed every month with their reviews, companies observed every quarter with their headcount. Collect the same entities on a schedule and you have a panel. For the cleaning steps that come before, see data wrangling in Python; for prices and quotes over time, see scraping Google Finance and financial data; and for the business side, market research data.


