Working with Apache Iceberg Tables in Spark

In this part, we will use the interactive shell for hands-on practice.

Of course, this is not how you would usually build a production Spark application. In real systems, you would write application code, manage dependencies, and run jobs in a more structured way. But for learning and experimenting, the interactive shell is a great tool. It lets you try things quickly and see the results immediately.

In this part, our goal is simple: we will create and update Apache Iceberg tables using Spark.

Note To use Iceberg in Spark, Spark must be configured to connect to an Iceberg catalog with a catalog name. The pyspark.sh interactive shell wrapper does that for you and connects to Oleander’s ready-to-use Iceberg catalog named oleander.

Creating a Namespace

Let’s start by creating a namespace.

This is very simple:

spark.sql("CREATE NAMESPACE oleander.tutorial")

Now you have a namespace called tutorial in the oleander Iceberg catalog.

Sometimes you may not know whether the namespace already exists, but you still want to make sure it is there. In that case, you can use IF NOT EXISTS:

spark.sql("CREATE NAMESPACE IF NOT EXISTS oleander.tutorial")

That way, the command succeeds whether the namespace is already there or not.

Creating a Table

A namespace is just a logical grouping of tables, so it only needs a name.

A table needs more information. To create a table, you need to define what kind of data it will store. In other words, you need to define its schema.

Let’s create a users table:

spark.sql("""
    CREATE TABLE IF NOT EXISTS oleander.tutorial.users (
      id STRING,
      email STRING,
      name STRING,
      joined_at TIMESTAMP
    )
    USING iceberg
""")

Now let’s verify that the table was created:

spark.sql("SHOW TABLES IN oleander.tutorial").show()

You should see something like this:

+---------+---------+-----------+
|namespace|tableName|isTemporary|
+---------+---------+-----------+
| tutorial|    users|      false|
+---------+---------+-----------+

You can also inspect the table schema with:

spark.sql("DESCRIBE TABLE oleander.tutorial.users").show(truncate=False)

The result should look like this:

+---------+---------+-------+
|col_name |data_type|comment|
+---------+---------+-------+
|id       |string   |NULL   |
|email    |string   |NULL   |
|name     |string   |NULL   |
|joined_at|timestamp|NULL   |
+---------+---------+-------+

At this point, you have created your first Iceberg table from Spark.

Altering a Table

It would be great if we perfectly knew what columns were needed for the business when creating a table, but that might only happen in an alternate universe.

In our universe, unfortunately, human beings are incomplete and we do not know the future. Business requirements change over time, and our tables need to evolve too. That might mean adding columns, renaming columns, or dropping columns.

That is exactly why schema evolution matters.

Adding a Column

Let’s say we want to add a Boolean column called active to indicate whether a user is active.

spark.sql("""
    ALTER TABLE oleander.tutorial.users
    ADD COLUMN active BOOLEAN
""")

Now check the schema again:

spark.sql("DESCRIBE TABLE oleander.tutorial.users").show(truncate=False)

You should see:

+---------+---------+-------+
|col_name |data_type|comment|
+---------+---------+-------+
|id       |string   |NULL   |
|email    |string   |NULL   |
|name     |string   |NULL   |
|joined_at|timestamp|NULL   |
|active   |boolean  |NULL   |
+---------+---------+-------+

Renaming a Column

Adding a column is only one kind of schema change.

Suppose you add active, but then your coworker suggests that is_active is a clearer name. Since Iceberg is designed with schema evolution in mind, this is also a straightforward operation.

spark.sql("""
    ALTER TABLE oleander.tutorial.users
    RENAME COLUMN active TO is_active
""")

Now verify the schema again:

spark.sql("DESCRIBE TABLE oleander.tutorial.users").show(truncate=False)

You should see:

+---------+---------+-------+
|col_name |data_type|comment|
+---------+---------+-------+
|id       |string   |NULL   |
|email    |string   |NULL   |
|name     |string   |NULL   |
|joined_at|timestamp|NULL   |
|is_active|boolean  |NULL   |
+---------+---------+-------+

Dropping a Column

After all that discussion about naming, the team might eventually decide that it does not actually need to store whether a user is active at all.

In many systems, it is often acceptable to leave an unused column in place. But unused columns can still add cognitive overhead, especially for people who are new to the codebase or the data model.

And, more importantly for us, this is a tutorial, so it is a good chance to learn how to remove a column.

Here is how to drop the column:

spark.sql("""
    ALTER TABLE oleander.tutorial.users
    DROP COLUMN is_active
""")

Now if you describe the table again:

spark.sql("DESCRIBE TABLE oleander.tutorial.users").show(truncate=False)

you should see that is_active is gone:

+---------+---------+-------+
|col_name |data_type|comment|
+---------+---------+-------+
|id       |string   |NULL   |
|email    |string   |NULL   |
|name     |string   |NULL   |
|joined_at|timestamp|NULL   |
+---------+---------+-------+

Deleting a Table

A table is not just a name in the catalog. It is where data actually lives, and it consumes storage.

In the real world, many teams end up paying storage costs for tables they no longer use. That is wasteful, and over time it can add up.

Fortunately, deleting a table is simple:

spark.sql("DROP TABLE IF EXISTS oleander.tutorial.users")

Now confirm that the table is gone:

spark.sql("SHOW TABLES IN oleander.tutorial").show()

You should see an empty result:

+---------+---------+-----------+
|namespace|tableName|isTemporary|
+---------+---------+-----------+
+---------+---------+-----------+

One small note, though: in real production systems, the hard part is usually not how to delete a table. The hard part is figuring out which tables are truly no longer being used.

Summary

In this part, we started working with Apache Iceberg tables from Spark.

Using the interactive shell, we created a namespace, created a table, inspected its schema, and then walked through a few common schema changes: adding a column, renaming a column, and dropping a column. Finally, we deleted the table.

These are some of the most basic and useful operations when getting started with Iceberg in Spark. In the next part, we can move on to actually inserting, reading, and querying data in Iceberg tables.