A data pipeline is an automated series of steps that moves data from where it is created to where it is used — ingesting from sources, transforming it into a usable shape, storing it, and serving it to the dashboards, models or apps that consume it. Every pipeline, from a nightly report job to a real-time fraud system, is a variation on those four stages plus an orchestrator that runs them reliably.
The four stages, in order
| Stage | What happens | Example |
|---|---|---|
| Ingest | Pull or receive raw data from sources | APIs, databases, files, scraped web pages, event streams |
| Transform | Clean, validate, join, reshape | Dedupe, fix types, enrich, aggregate |
| Store | Land the result somewhere queryable | Warehouse, lake, database |
| Serve | Deliver to consumers | BI dashboards, ML training, application queries |
As IBM's and AWS's explainers both frame it, the pipeline's job is to make this repeatable and trustworthy — so the data arriving at the "serve" end is fresh, correct and shaped the way consumers expect, without a human copying files around.
ETL vs ELT: when transformation happens
The classic debate is really about ordering. ETL (Extract, Transform, Load) cleans and reshapes data before loading it into storage — the transform runs on a separate engine, and only finished data lands. ELT (Extract, Load, Transform) loads raw data first, then transforms it inside the warehouse using its compute. ELT has become the default for cloud warehouses because storage is cheap and warehouse compute is powerful, so keeping the raw data and transforming on demand is flexible. ETL still wins when you must not store sensitive raw data, or when the transformation is too heavy for the warehouse. Neither is universally right; the choice follows your storage cost, compliance constraints and transform complexity.
Batch vs streaming: when data moves
The other axis is timing. Batch pipelines process data in chunks on a schedule — every hour, every night — and are simpler, cheaper and perfectly adequate for most reporting and analytics. Streaming pipelines process each event as it arrives, for use cases where minutes-old data is too stale: fraud detection, live dashboards, real-time personalization. Streaming costs more to build and operate, so the honest default is batch unless the business genuinely needs sub-minute freshness. Many real systems are hybrid — streaming for the time-critical path, batch for everything else.
Orchestration: the part that makes it a pipeline
A sequence of scripts is not a pipeline until something runs them reliably. Orchestrators — Airflow, Dagster, Prefect and the managed equivalents — schedule the stages, enforce their order and dependencies, retry failures, and alert when something breaks. This is where a pipeline earns its name: it handles the ugly reality that sources go down, data arrives late or malformed, and a step that worked yesterday fails today. Good orchestration turns "the report is wrong and nobody noticed" into "the job failed at 2am and paged the on-call". Underneath, the non-negotiables are the same across tools: idempotent steps (re-running does not double-count), data validation at boundaries, and observability so you can answer "is today's data correct?" without spelunking through logs.
Where web data enters the pipeline
For a growing share of pipelines, one of the most valuable sources sits at the ingest stage: the open web. Competitor prices, market listings, company data, reviews and public documents are data your internal systems do not have, and they feed pricing models, market dashboards, lead databases and RAG stores. The challenge is that the web is a hostile source — pages block bots, render in JavaScript, and change structure — so "ingest from the web" is its own engineering problem inside the wider pipeline. A scraping API that returns clean Markdown or structured JSON turns that hostile source into a well-behaved one: your ingest step makes an API call and gets back consistent, parseable data, the same way it would from any other source. For discovering what to ingest across a whole site, a crawl or map call enumerates the pages; for agent-driven pipelines, the same capabilities are MCP tools. We go deeper on the collection side in how to get data for AI.
Designing one that lasts
The pipelines that survive share a few habits regardless of tools: they validate data at every boundary rather than trusting sources, they make each step idempotent so a retry is safe, they keep raw data so a transform bug can be fixed by reprocessing rather than re-collecting, and they monitor data quality — freshness, row counts, null rates — not just whether the job exited zero. Start with the simplest thing that works (a scheduled batch job beats a streaming system you do not need), add complexity only when a real requirement forces it, and treat the ingest sources — especially the web ones — as unreliable by default. A pipeline is only as trustworthy as its least reliable source, so hardening ingest is usually where the effort pays off most.