Ollie
In the previous part, we looked at the two basic building blocks of data systems: storage and compute.
Now it is time to talk about Apache Iceberg.
Iceberg is a big project, and there are many engineering details behind it. In this tutorial, though, we are not going to start with the internals. Instead, we will focus on the main ideas you need in order to understand where Iceberg fits, why it exists, and why so many modern data systems use it.
What Is Apache Iceberg?
At the simplest level, Apache Iceberg is an open table format for large analytical datasets.
That definition is short, but it helps to unpack it a little.
First, Iceberg is a table format. That means it defines how a table is represented and managed. It describes things like which data files belong to the table, how changes to the table are tracked, and how a system should understand the table as a whole.
Second, Iceberg is open. It is not tied to one vendor or one processing engine. Different tools can implement Iceberg support and work with the same tables.
Third, Iceberg is designed for large analytical datasets. It is meant for data systems where tables can become very large, keep growing over time, and be accessed by different tools for analytics and data processing.
It is also helpful to be clear about what Iceberg is not.
Iceberg is not a database by itself. It does not store data by replacing object storage or a filesystem. It is not a compute engine either. It does not process your data on its own. Instead, it defines how tables are organized so that compute engines such as Apache Spark can read and write them reliably.
So when people say they are using Iceberg, what they usually mean is that their table data is stored in places like S3, and Iceberg is the format that makes those files behave like proper analytical tables.
Why Do We Need It?
At first, object storage can seem simple enough. You can store files in S3, organize them into folders, and point your processing engine at them. For small or simple use cases, that may be enough.
But as systems grow, raw files alone stop being enough.
You want your data to behave like tables, not just like a pile of files. You want to know which files belong to a table. You want to change schemas without breaking everything. You want partitioning to help performance without becoming a constant source of pain. You want reads and writes to stay reliable even as data grows. And if multiple jobs interact with the same table, you want those changes to be managed safely.
Older data lake approaches could work, but they often came with awkward limitations. Managing partitions could be clumsy. Schema changes could be risky. Keeping track of table state could become fragile. And as workloads grew, those rough edges became more painful.
Iceberg exists to solve that kind of problem.
It gives you a way to manage large analytical tables on top of object storage, while keeping the table behavior much closer to what users expect from a modern data system. Instead of treating data as scattered files that users must reason about directly, Iceberg gives systems a structured way to understand the table as one logical thing.
That is the key idea: your data may physically live as files, but users and engines work with it as a table.
Where Does It Fit in the Stack?
By now, you have probably seen the shape of the stack emerging.
At the bottom, you have storage, often object storage such as Amazon S3. That is where the actual data files live.
Above that, you have Apache Iceberg, which defines how those files are organized and tracked as tables.
Then you have a compute engine, such as Apache Spark, which reads and writes those Iceberg tables.
So Iceberg sits in the middle. It is the table layer between storage and compute.
That position is important. Iceberg does not replace storage, and it does not replace compute. Instead, it gives both sides a shared understanding of what the table is.
Without something like Iceberg, a compute engine may still be able to read files from storage, but it has much less help in understanding table structure, versions, schema changes, and safe updates. With Iceberg, the compute engine can interact with data at the table level instead of treating everything as loose files.
That is one of the reasons Iceberg is such a useful concept for beginners to learn. It helps separate three things that are easy to blur together:
- where the data is stored
- how the data is organized as a table
- which engine is used to process it
In this tutorial series, those will usually mean:
- S3 for storage
- Apache Iceberg for the table format
- Apache Spark for compute
What Problems Does It Solve?
Iceberg solves several practical problems that show up in modern data systems.
One is table management at scale. When a table grows large, it may be backed by many files. You do not want users or applications to manually track which files belong to which version of the table. Iceberg handles that through metadata.
Another is schema evolution. Real data changes over time. Columns are added, renamed, or removed. A good table format needs to support those changes in a controlled way.
Another is partitioning. Partitioning helps performance, but older approaches often made it too visible and too fragile. Iceberg improves this by making partition handling more table-driven and less dependent on users manually managing directory layouts.
Another is reliable writes. In a modern data system, writing a table is not just about dropping files into storage. The system also needs to safely update the table’s metadata so readers see a consistent state.
And another is interoperability. Because Iceberg is an open format, different engines can work with the same table instead of locking you into one proprietary system.
You do not need to memorize all of these on day one. The important thing is to see the overall pattern: Iceberg solves the gap between raw files and usable analytical tables.
What Makes It Powerful?
One of the reasons Iceberg is widely adopted is that it brings several features together in a way that is practical for real systems.
A very important one is snapshot-based table state.
When a table changes, Iceberg tracks that change as a new snapshot. You can think of a snapshot as a recorded version of the table at a point in time. It represents which data files make up the table state for that version.
This idea is powerful because it means a table is not just “whatever files happen to be there right now.” Instead, the table has explicit versions, and the current version is well defined.
That leads directly to another important feature: time travel.
Because Iceberg keeps snapshots, a system can read an older version of the table. That is what people mean by time travel. You are not literally traveling through time, of course. You are asking the system to read the table as it existed at an earlier snapshot.
This is useful for several reasons. It can help with debugging. It can help with reproducibility. It can help you understand what changed between versions of a dataset. And in some cases, it can help recover from mistakes by letting you inspect an earlier state.
Even without going deep into the internals, this gives you a strong mental model:
- the table data lives in files
- Iceberg metadata tracks those files
- a snapshot represents one version of the table
- time travel means reading an older snapshot
Iceberg also supports other useful features, such as schema evolution, partition evolution, and better interoperability across engines. But for a beginner, the snapshot idea is one of the best starting points because it explains why Iceberg tables feel more like real managed tables and less like a random collection of files.
What Iceberg Is Not
It is worth repeating what Iceberg does not do, because beginners often mix these roles together.
Iceberg is not a compute engine. It does not execute transformations, run SQL by itself, or process data on its own. You still need something like Apache Spark, DuckDB, Trino, Flink, or another engine to actually do work with the table.
Iceberg is not object storage. Your data files still need to live somewhere, such as S3, ADLS, or another storage system.
Iceberg is not a magic solution for every data problem. It does not automatically make a bad pipeline good, and it does not remove the need for careful data modeling, performance tuning, or operational discipline.
What it does do is provide a strong, open foundation for managing analytical tables in a modern data system.
That is why it matters.
Summary
In this part, we introduced Apache Iceberg as an open table format for large analytical datasets.
We looked at why it exists, where it fits in the stack, and what kinds of problems it solves. Most importantly, we introduced one of Iceberg’s core ideas: snapshots. An Iceberg table is not just a set of files. It has a tracked table state, and each change creates a new snapshot. Because of that, systems can also support time travel, which means reading older versions of the table.
That is enough for now.
In the next part, we will switch to the other side of the picture and talk about Apache Spark, the compute engine we will use throughout the rest of this tutorial series.