Ollie
In the previous part, we introduced Apache Iceberg as the table layer that sits between storage and compute.
Now it is time to look at the compute side.
Apache Spark is one of the most widely used compute engines in modern data systems, especially when the job involves large amounts of data. People use it to process data in batches, transform and move data between systems, analyze large datasets, and prepare data for downstream use. But before getting into how to write Spark code or how to tune Spark jobs, it helps to first understand what Spark actually is, what it is for, and why it has become such a central tool in the data ecosystem.
In this part, we will stay at that higher level.
What Is Apache Spark?
At the simplest level, Apache Spark is a distributed compute engine for large-scale data processing.
That sentence carries a lot, so it is worth unpacking.
First, Spark is a compute engine. Its job is to process data. It reads data from somewhere, performs work on it, and writes results somewhere else.
Second, Spark is distributed. That means it is designed to split work across multiple machines, or at least multiple execution units, so that it can process more data than a single machine could handle comfortably on its own.
Third, Spark is built for large-scale data processing. It is commonly used when the amount of data is too large for a single machine to process efficiently, or when the workload benefits from being spread across multiple machines.
Spark gives you a general engine for working with data at scale. You can use it to filter, join, aggregate, transform, and write data. You can use SQL with it. You can use DataFrame APIs with it. You can run it on your laptop for small tests, or on a cluster for much larger workloads.
That flexibility is one of the reasons Spark is so widely adopted.
What Is It For?
Spark is for processing data.
That may sound obvious, but it is useful to be more concrete. Spark is commonly used for things like:
- reading data from storage
- cleaning or reshaping that data
- joining datasets together
- computing aggregates and metrics
- writing processed results back to storage
- running recurring data pipelines
- preparing data for analytics, reporting, or machine learning
In many real systems, Spark is the engine that turns stored data into useful data.
For example, a company may ingest logs, events, transactions, or application records into storage. Spark jobs can then clean that data, reshape it, enrich it, combine it with other datasets, and write it into analytical tables for reporting or downstream use.
That is the general role Spark plays in a data system. It is not usually the place where the data permanently lives. It is the engine that does work on the data.
In the context of this tutorial series, that means Spark will be the engine we use to read and write Apache Iceberg tables.
Where Does Spark Fit in the Stack?
By now, the overall shape of the system should feel familiar.
At the bottom, you have storage, such as S3, where data files live.
In the middle, you may have a table format, such as Apache Iceberg, which organizes those files into tables.
Then you have Apache Spark, which reads from those tables, processes the data, and writes results back.
So Spark sits on the compute side of the stack.
That distinction matters because beginners often blur these roles together. Spark is not where the data fundamentally lives, and Spark is not the definition of the table format. Spark is the engine that performs the work.
That is also why Spark pairs so naturally with systems like Iceberg. Iceberg defines the table. Spark performs computation against that table.
What Makes Spark Powerful?
Spark is powerful for a few different reasons.
One major reason is scale. Spark was designed for workloads that may be too large for a single machine to handle comfortably. By distributing work across many executors, Spark can process large datasets more effectively than a single-node tool in those cases.
Another reason is a relatively high-level programming model. Users do not usually have to manually coordinate every detail of distributed execution. Instead, they describe transformations on data, and Spark builds and executes a plan for doing that work.
That makes a big difference. Distributed systems are hard. Spark does not remove that complexity completely, but it gives users a much more approachable way to work with distributed data processing.
Another reason is versatility. Spark supports multiple ways of interacting with data. You can use SQL. You can use DataFrames. You can use lower-level APIs when needed. It can be used for batch processing, and it also has support for structured streaming.
Spark is also powerful because it works with many data systems. It can read from and write to many storage systems, table formats, and data sources. That makes it practical in real environments, not just interesting in theory.
And finally, Spark is powerful because it has become a kind of standard tool. That means there is a large community, broad documentation, many integrations, and a lot of production experience behind it. For a tutorial series, that matters. It makes Spark a very useful engine to learn because the ideas transfer widely across the data ecosystem.
What Spark Is Not
It is just as important to understand what Spark is not.
Spark is not a data warehouse. It is not the system where your analytical tables permanently live.
Spark is not a table format. It does not define how a table is represented in the way Apache Iceberg does.
Spark is not object storage. It still needs storage systems such as S3, HDFS, or local filesystems to read from and write to.
Spark is not a magic solution for every data problem. Just because Spark can process large datasets does not mean it is always the right choice. If your data is small enough, a simpler tool may be cheaper, easier, and faster.
This is worth stressing because many beginners hear “big data” and immediately assume Spark is always the answer. That is not true.
If a workload fits comfortably on a single machine, tools like DuckDB can often be a better choice. They are simpler to operate and may perform very well for smaller workloads. Spark becomes more compelling when scale, distribution, and integration with larger data platforms start to matter.
So the value of Spark is not that it replaces every other tool. The value is that it is a strong answer for a specific class of data processing problems.
Why We Use Spark in This Tutorial
This tutorial series is called Apache Iceberg & Spark 101, so Spark will be the compute engine we focus on throughout the rest of the series.
We are using Spark for a few reasons.
First, it is widely adopted. If you spend enough time around modern data systems, you will see Spark everywhere.
Second, it is highly capable. Spark can handle a wide range of practical data processing tasks, from straightforward transformations to large distributed jobs.
Third, Spark works very naturally with Apache Iceberg. That makes it a strong fit for a tutorial series that wants to explain both the table side and the compute side of a modern data system.
And finally, Spark is worth learning because many people use it without fully understanding where it fits or how to think about it. Learning Spark from the right mental model makes the rest of the ecosystem easier to understand too.
Summary
In this part, we introduced Apache Spark as a distributed compute engine for large-scale data processing.
We looked at what Spark is, what it is for, where it fits in the stack, and why it is powerful. Most importantly, Spark is the engine that does work on data. It reads from storage or tables, processes data, and writes results back. In this tutorial series, it will be the compute engine we use alongside Apache Iceberg.
That is enough for now.
In the next parts, we will move from concepts to practice and start looking at how Spark actually works with data and tables in real workflows.