Ollie's Growing Tips
Apache Iceberg & Spark 101
17 articles · ~2h 46m · 227 snippets
[0]Foundations
Storage, compute, and where Iceberg and Spark fit.
[0]
The Basic Building Blocks of Data Systems
Every garden needs soil and sunlight. Every data system needs storage and compute. Learn the two pieces everything else grows from.
~6 min read
[1]
Introduction to Apache Iceberg
Apache Iceberg is the raised bed of the data garden: an open table format that keeps files in object storage organized as tidy, reliable tables.
~8 min read
[2]
Introduction to Apache Spark
Apache Spark is the heavy machinery of the plot: a distributed compute engine that works your data at any scale.
~7 min read
[1]Getting started
Core concepts, a working Spark setup, and your first Iceberg tables.
[3]
Apache Iceberg: Concepts and Terminology
Catalogs, snapshots, schemas, and partitions. Learn the botanical names of your Iceberg garden so you can talk shop with any gardener.
~7 min read
[4]
Apache Spark: Getting Started
Set up Spark locally, start a session, and run your first commands against an Iceberg catalog.
~5 min read · 11 snippets
[5]
Working with Apache Iceberg Tables in Spark
Create, load, update, and inspect Iceberg tables from Spark. Daily care for healthy tables.
~5 min read · 19 snippets
[6]
Apache Spark: Concepts and Terminology
Drivers, executors, jobs, stages, and tasks. How Spark actually divides up the work in the field.
~6 min read · 1 snippet
[2]Analyzing clickstream data
Summaries, funnels, and joins over a retail clickstream table.
[7]
Setting Up a Retail Clickstream Table with Apache Iceberg
Break ground on the retail clickstream table we will cultivate for the rest of the series.
~2 min read · 4 snippets
[8]
Summarizing Clickstream Events with Spark
Turn raw clickstream events into daily summaries with Spark SQL and the DataFrame API. Your first basket of produce.
~17 min read · 26 snippets
[9]
Analyzing Product Funnels with Aggregations
Follow sessions from product view to purchase with aggregations, the way a gardener tracks seed to fruit.
~13 min read · 17 snippets
[10]
Measuring Cart Abandonment with Joins
Some carts never ripen into purchases. Use joins to find what wilted on the vine and when.
~13 min read · 19 snippets
[11]
Understanding Join Types in Spark
Inner, left, full, semi, and anti joins. Choosing the right graft when joining two tables together.
~13 min read · 30 snippets
[3]Production techniques
Retry-safe writes, partitioning, time travel, streaming, and compaction.
[12]
Making Batch Pipelines Retry-Safe with MERGE INTO
Retries happen. MERGE INTO keeps a re-run pipeline from double-planting the same rows.
~14 min read · 22 snippets
[13]
Improving Query Performance with Partitioning
Partition your tables so queries only walk the rows they need instead of trampling the whole field.
~9 min read · 10 snippets
[14]
Querying Historical Data with Time Travel
Iceberg keeps a journal of every change. Query your table as it was, compare seasons, and roll back mistakes.
~10 min read · 21 snippets
[15]
Streaming Clickstream Events into Iceberg
Swap the watering can for drip irrigation: stream clickstream events into Iceberg with Structured Streaming.
~18 min read · 29 snippets
[16]
Managing Small Files with Compaction
Frequent commits leave lots of tiny files. Prune them back with compaction so the table stays healthy as it grows.
~13 min read · 18 snippets
You are loved. Always water & happy growing...