Apache Spark: Concepts and Terminology

In the previous parts of this tutorial series, we set up a workspace for Spark, started an interactive shell, and used it to work with Iceberg tables. Along the way, we used the pre-initialized SparkSession object called spark, but we have not really stopped to explain what that object is or how it fits into Spark as a whole.

Before going further, it is a good time to pause and learn some of the basic concepts and terminology of Apache Spark.

Spark has a lot of terms, and at first they can feel a little overwhelming. The good news is that you do not need to understand every internal detail to use Spark effectively. What matters first is building a clear mental model of the main pieces and how they relate to each other.

Spark Execution Environment

Spark Cluster

A Spark cluster is the environment where Spark runs distributed computation.

If you run Spark on a single machine, the setup can feel very simple. But Spark is designed to scale beyond one machine. When it does, it needs a collection of machines working together. That collection is what we call a Spark cluster.

The exact shape of the cluster depends on the platform and deployment model. Sometimes you manage the infrastructure yourself. Sometimes a managed platform does it for you. Either way, the basic idea is the same: Spark uses a cluster so it can distribute work across multiple machines.

For a beginner, the most important thing to remember is this:

A Spark cluster is the computing environment that Spark uses to run jobs.

Driver

The driver is the main process that coordinates a Spark application.

When you write Spark code, you are usually writing logic that runs in the driver process. The driver is responsible for creating the Spark session, building the execution plan, coordinating execution, and collecting results when needed.

The driver does not usually do all of the heavy data processing itself. Instead, it coordinates the application and tells the rest of Spark what work needs to be done.

If you are using the interactive shell, the shell you are typing into is connected to the driver.

Executors

Executors are the processes that actually carry out the distributed work.

When Spark needs to process data in parallel, it sends work to executors. The executors run tasks, read data, perform transformations, and return results or intermediate outputs as needed.

In a real distributed job, there are usually multiple executors working at the same time. That is one of the main reasons Spark can handle large-scale data processing.

A simple mental model is this:

• the driver coordinates the application
• the executors perform the distributed work

That is not the whole story, but it is the right place to start.

Spark Entry Points and Core Objects

SparkSession

A SparkSession is the main entry point for working with Spark.

Through SparkSession, you get access to Spark’s higher-level APIs, including DataFrames and SQL.

That is why in the interactive shell you can write code like this:

spark.sql("SELECT 'Hello, Spark!' AS greeting").show()

Here, spark is the SparkSession.

For a beginner, the easiest way to think about it is this:

SparkSession is the main object you use to interact with Spark.

If you are writing Spark code today, this is the object you will most often work with directly.

SparkContext

The SparkContext is a lower-level core object in Spark.

Historically, it was the main entry point to Spark. Today, in most applications, you usually access Spark through SparkSession instead. In fact, SparkSession sits above it and uses it underneath.

So why does SparkContext still matter?

Because it is still a core part of Spark’s architecture, and you will see it mentioned in documentation, logs, and some code examples. It represents the connection between your application and the Spark runtime and is involved in lower-level coordination of execution.

For most beginners, the important point is simply this:

You will usually work with SparkSession, but SparkContext is one of the lower-level objects underneath it.

You do not need to use it directly very much at the start.

Spark Execution Model

Stage

A stage is a group of tasks that Spark can execute together as part of a job.

When you run a Spark operation, Spark does not usually execute it as one giant block of work. Instead, it breaks the work into smaller pieces. One level of that breakdown is the stage.

A stage usually represents a chunk of computation that can be run in parallel before Spark needs to reshuffle or reorganize data for the next step.

You do not need to know all the details of stage planning yet. For now, it is enough to know that Spark breaks jobs into stages as part of its execution plan.

Task

A task is the smallest unit of work that Spark sends to an executor.

If a stage is a chunk of work, tasks are the individual work items inside that chunk.

For example, if Spark wants to process many partitions of data in parallel, it may create many tasks and distribute them across executors. Each task works on a piece of the data.

A simple way to think about the relationship is this:

• a job is broken into stages
• a stage is made up of tasks
• tasks are run by executors

That mental model is enough for now.

Spark Data Abstractions

DataFrame

A DataFrame is Spark’s main high-level abstraction for working with structured data.

If you have used tables in SQL, the idea will feel familiar. A Spark DataFrame has rows and columns, and it is designed for operations such as selecting columns, filtering rows, joining datasets, and aggregating values.

This matters especially for this tutorial series because we are mainly working with Apache Iceberg, which stores structured tables. That means the DataFrame is the abstraction we will use most throughout the tutorial. When we read an Iceberg table in Spark, transform it, or write it back, we will usually be working with DataFrames.

For beginners, this is the most important Spark data abstraction to get comfortable with.

RDD

An RDD, short for Resilient Distributed Dataset, is an older and lower-level Spark abstraction. It represents distributed data that Spark can process in parallel.

RDDs are still part of Spark, and they are important historically, but they are not where most beginners should start.

For this tutorial series, the main thing you need to know is simple:

RDD exists, but we will mostly work with DataFrames instead.

That is because DataFrames are a better fit for structured data, and structured data is exactly what we are dealing with when working with Iceberg tables.

Summary

In this part, we covered some of the most important Apache Spark concepts and terms:

• Spark cluster
• driver
• executors
• SparkSession
• SparkContext
• stage
• task
• DataFrame
• RDD

You do not need to memorize every detail right away. What matters most is building a usable mental model.

Spark runs in a cluster. The driver coordinates the application. Executors perform the distributed work. SparkSession is the main object you use to interact with Spark. SparkContext is a lower-level core object underneath it. Spark breaks work into stages and tasks. And when working with structured data, the main abstraction we will use throughout this tutorial is the DataFrame, especially because we are working with Iceberg tables.