Apache Spark: Getting Started

In the previous parts of this tutorial series, we introduced Apache Iceberg and covered some of its basic concepts and terminology. At this point, you may be ready to start working with Iceberg tables.

But as we mentioned earlier, Apache Iceberg is not a compute engine. It defines how tables are organized and managed, but it does not process data by itself. To actually create tables, read from them, write to them, and run queries against them, we need a compute engine.

Since this is Apache Iceberg & Spark 101, the compute engine we will use is Apache Spark.

In this part, we will take a more hands-on step and get started with Spark.

Using Oleander

To run Spark jobs in a real environment, you need somewhere to run them. In production, that usually means submitting jobs to a Spark cluster. There are many platforms that support Spark, and you may choose one of them or manage your own Spark infrastructure.

In this tutorial, we will use Oleander to manage and run Spark jobs.

Oleander provides a fully managed environment with Spark and a ready-to-use Iceberg warehouse. It is designed so that you can start writing and submitting Spark jobs without having to set up all of the infrastructure yourself first. Oleander also provides a CLI tool, oleander-cli, which we will use throughout this tutorial to interact with the platform.

You can sign up at https://oleander.dev. You do not need a credit card to get started, and the Hobby plan comes with $5 of credit so you can try it out.

For more information, visit https://oleander.dev.

Installing and Configuring oleander-cli

We will use oleander-cli to interact with Oleander and run Spark jobs.

On macOS, you can install it with Homebrew:

brew tap OleanderHQ/tap
brew install oleander-cli

Once it is installed, you need to configure it with your API key.

From the oleander.dev website, go to Settings → API keys. You should see a default API key already created for you. Copy that key and run:

oleander configure --api-key <your-api-key>

After that, your CLI is ready to use.

Initializing a Spark Project

Now we need some Spark code to run.

Writing a minimal runnable Spark application is not very complicated, but for beginners it can still feel like extra setup work before getting to the interesting part. To make that easier, Oleander provides a PySpark project template that you can initialize directly from the CLI.

For example:

oleander spark init spark-tutorial-app

This creates a directory called spark-tutorial-app and initializes a basic runnable Spark application inside it.

If you already created a directory and want to initialize the project there, run this inside that directory instead:

oleander spark init .

During initialization, the CLI will ask for two inputs:

• the entrypoint file name
• the custom package name

The entrypoint file is the main file for your Spark application. The custom package is where you can put helper code or reusable modules that your application imports.

For this tutorial, I will use:

• tutorial.py for the entrypoint file
• tutorial_lib for the custom package name

You will see something like this:

┌  Initialize Spark project
│
◇  Template extracted
? Put spark entrypoint filename to use: tutorial.py
? Put custom package name to use: tutorial_lib
│
◇  Spark template initialized

At this point, you have a minimal Spark project scaffolded and ready to work with.

Setting Up the Python Environment

Note Spark also supports other languages such as Scala and R, but throughout this tutorial series we will use Python.

The initialized PySpark project uses uv for Python project and package management.

uv is a Python package and project manager written in Rust. If you do not already have it installed, refer to the installation guide here: https://docs.astral.sh/uv/getting-started/installation/

Once uv is installed, go into your project directory and run:

uv sync

This creates a virtual environment for the project and installs the dependencies needed for local development, including pyspark.

Then activate the virtual environment:

source .venv/bin/activate

Now you have a minimal working Spark application environment on your machine.

Running Your First Spark Application

The generated project includes a helper script called spark-submit.sh.

To test your application, run:

./spark-submit.sh tutorial.py

This runs the example Spark application from your local machine so you can verify that everything is working.

That is enough to get started. You now have a minimal runnable Spark application.

Using the PySpark Shell

When working in Python, it is often convenient to test ideas in an interactive shell instead of writing a temporary script every time. Spark supports that style too.

The generated project includes a helper script called pyspark.sh that starts an interactive PySpark shell with Spark and Iceberg already initialized.

Run:

./pyspark.sh

This opens a Python shell where you can immediately execute Spark code.

For example, try this:

spark.sql("SELECT 'Hello, Spark!' AS greeting").show()

You should see output like this:

+-------------+
|     greeting|
+-------------+
|Hello, Spark!|
+-------------+

The spark object is a pre-initialized Spark session. You do not need to fully understand Spark sessions yet; we will cover that soon. For now, the important point is that you already have a working environment where you can run Spark SQL right away.

Summary

In this part, we got started with Apache Spark in a practical way.

We looked at how to use Oleander as the environment for this tutorial, how to install and configure oleander-cli, how to initialize a minimal PySpark project, and how to run both a Spark application and an interactive PySpark shell.

You may have noticed that I mentioned Oleander comes with a ready-to-use Iceberg setup, but we have not actually done anything with Iceberg yet.

That is intentional.

Before using Iceberg tables, we first needed a working compute environment. Now we have one. In the next part, we will start using Spark to actually interact with Iceberg tables.