Ollie
Welcome to the world of data systems.
You may or may not already know a little about data processing. You may or may not have worked with a modern data stack before. Either way, that is completely fine. If you are new to this field, this tutorial will give you some ground to stand on. If you already have some experience, it should still serve as a useful recap.
This is the very first article in Apache Iceberg & Spark 101. Before we get into what Apache Iceberg is and what Apache Spark is, it helps to first understand the bigger picture. At a very high level, data systems are built from two basic pieces: storage and compute.
Storage answers the question, where does the data live?
Compute answers the question, how do we process that data?
That is the foundation. Once those two pieces make sense, Apache Iceberg and Apache Spark will make much more sense too.
Storage
From old systems to modern ones, one of the most fundamental parts of a computer program is I/O: input and output. A program reads input and writes output. Data systems are not much different. They read data and write data.
The big difference is scale.
In data systems, especially modern ones, we often deal with much larger volumes of data, and that data usually keeps growing over time. Fortunately, cloud infrastructure is now common, and object storage such as Amazon S3 gives us a highly scalable place to store large amounts of data.
But raw storage alone is not enough.
Before going further, it is worth making one thing clear: not every system needs cloud-scale storage. Your data might be small enough to fit on a modern HDD or SSD. Your team might already share files through a NAS. And if that setup is enough for your business, that is great. You should stick with it. It is always better to use the right level of technology than to adopt something more complicated just because it sounds modern.
For the rest of this tutorial, though, let’s assume your data keeps growing over time and needs a storage layer that can grow with it. In that world, cloud object storage such as S3 becomes a practical choice.
Even then, you still need something more.
You do not want your data to exist only as raw files. Business data serves many different purposes, and because of that, it comes in many different structures. You do not want to parse raw text or binary files every time you need information. You do not want a completely different way to read every dataset. You want structured data that tools can understand consistently.
This is not a new idea. We have been using tabular data for decades, from spreadsheets like Lotus 1-2-3 to databases like PostgreSQL and beyond.
And again, if your data fits comfortably in a single PostgreSQL instance, and that works well for your needs, that is probably the best tool to use. PostgreSQL is not only excellent for transactional workloads, but also very capable for analytics. But if your data keeps growing, and a single database instance can no longer serve your needs or becomes too expensive, that is where more modern analytical storage approaches start to matter.
This tutorial series is called Apache Iceberg & Spark 101, so now is a good time to briefly introduce Apache Iceberg.
We will cover Iceberg in much more detail later. For now, the basic idea is enough.
In modern data systems, the actual data files often live in object storage such as S3. But users and processing engines are not supposed to manually manage those files one by one. There needs to be a shared way to describe tables: where they are, how they are organized, which files belong to them, how changes are tracked over time, and so on.
Apache Iceberg is an open table format that defines exactly that.
It is not a database by itself. It is not a compute engine either. It is a specification for how analytic tables should be represented and managed. Any system that understands the Iceberg format can read and write Iceberg tables.
Having a table format is not unique to Iceberg. PostgreSQL has its own internal format for storing tables. BigQuery has its own as well. The difference is that Iceberg is an open format. That means different tools and engines can implement it and work with the same tables. That openness is what gives you interoperability.
So when we talk about storage in this tutorial series, the basic idea is this: the data files live somewhere like S3, and Apache Iceberg provides an open way to organize those files as tables.
Compute
Once you have data stored in a structured tabular form, you still need something to actually do work with it.
You need a program that reads data, transforms it, analyzes it, and writes results back. That is the role of the compute engine.
“Compute engine” is a broad term, but in this tutorial, we will use it in a practical sense: a compute engine is a system designed to process data.
Because Apache Iceberg provides interoperability, you are not locked into one compute engine. You can choose the engine that fits your needs.
In this tutorial series, we will mainly focus on Apache Spark. That is because this is Apache Iceberg & Spark 101. But Spark is not the only option. If the data you want to process is small enough, DuckDB can be a great choice. It can be simpler, cheaper, and faster for many workloads. Apache DataFusion is also a strong option, especially if you want to build a more specialized processing system for your own product or business needs. Both can work with Apache Iceberg.
The important point is not that Spark is the only answer. The important point is that you need some kind of compute engine.
We chose Apache Spark for this tutorial because it is widely adopted, highly scalable, and very powerful for large-scale data processing. At the same time, many people find it difficult to use well. That makes it a good system to learn carefully and from first principles.
Putting Them Together
At a high level, a modern data system often looks like this:
- Storage is where the data lives.
- A table format defines how that stored data is organized as tables.
- Compute reads and writes those tables.
In the context of this tutorial series:
- object storage such as S3 stores the underlying data files
- Apache Iceberg defines the table format
- Apache Spark is the compute engine we will use to process the data
There are many alternatives in each category, but this series focuses on these two because they are practical, widely used, and a great way to understand modern data systems.
Summary
In this part 0, we covered the two key building blocks of data systems: storage and compute.
Storage answers where data lives. Compute answers how data gets processed.
We also briefly looked at where Apache Iceberg and Apache Spark fit into that picture. Iceberg gives us an open table format for organizing analytical data, and Spark gives us a powerful engine for processing it.
In the next parts, we will go deeper into each of them.