The anatomy of a data pipeline
Sources, ingestion, raw storage, transformation, serving — every data pipeline has the same five stages. This post maps them, explains what happens at each one, and shows where things go wrong.
Learn by doing
Sources, ingestion, raw storage, transformation, serving — every data pipeline has the same five stages. This post maps them, explains what happens at each one, and shows where things go wrong.
Three architectural approaches to storing and querying data at scale, each built to solve the problems of the one before it. Here is why lakehouses are where the industry is converging.
CSV, JSON, Parquet, Avro, ORC — what each format is optimised for and when to use it. Row vs columnar storage from first principles, with a clear decision rule for data engineering use cases.
Partitioning organises files by column value so queries can skip irrelevant data entirely. How partition pruning works, how to choose the right partition column, and why over-partitioning causes its own problems.
The techniques that make Parquet dramatically faster than CSV on the same hardware. Column pruning, row group statistics, bloom filters, and why sorting data within files amplifies all of them.
Open table formats add ACID transactions, time travel, and schema enforcement on top of object storage. What Delta Lake, Iceberg, and Hudi each bring, how they differ, and how to choose.