The anatomy of a data pipeline
Sources, ingestion, raw storage, transformation, serving — every data pipeline has the same five stages. This post maps them, explains what happens at each one, and shows where things go wrong.
Technology & Mathematics, charted
SkyDeLake is a field journal covering data engineering, generative AI, agentic systems, and mathematics — from school-level concepts to applied techniques — broken into write-ups and tutorials you can actually follow.
Latest
Sources, ingestion, raw storage, transformation, serving — every data pipeline has the same five stages. This post maps them, explains what happens at each one, and shows where things go wrong.
Three architectural approaches to storing and querying data at scale, each built to solve the problems of the one before it. Here is why lakehouses are where the industry is converging.
CSV, JSON, Parquet, Avro, ORC — what each format is optimised for and when to use it. Row vs columnar storage from first principles, with a clear decision rule for data engineering use cases.
Partitioning organises files by column value so queries can skip irrelevant data entirely. How partition pruning works, how to choose the right partition column, and why over-partitioning causes its own problems.
The techniques that make Parquet dramatically faster than CSV on the same hardware. Column pruning, row group statistics, bloom filters, and why sorting data within files amplifies all of them.
Open table formats add ACID transactions, time travel, and schema enforcement on top of object storage. What Delta Lake, Iceberg, and Hudi each bring, how they differ, and how to choose.
Learn by doing
Sources, ingestion, raw storage, transformation, serving — every data pipeline has the same five stages. This post maps them, explains what happens at each one, and shows where things go wrong.
Three architectural approaches to storing and querying data at scale, each built to solve the problems of the one before it. Here is why lakehouses are where the industry is converging.
CSV, JSON, Parquet, Avro, ORC — what each format is optimised for and when to use it. Row vs columnar storage from first principles, with a clear decision rule for data engineering use cases.
Browse
Pipelines, lakehouses, dbt, Spark, Kafka, and orchestration
Browse posts →ML, statistics, experimentation, and model evaluation
Browse posts →LLMs, prompt engineering, RAG, fine-tuning, and evaluation
Browse posts →Agentic systems, workflows, tool use, and orchestration
Browse posts →Algebra, calculus, statistics, and proofs — school to advanced
Browse posts →Tools, platforms, and general tech write-ups
Browse posts →