Managing Small Files with Compaction

[16]of 16

Ollie

head gardener

In Part 15, we built a Spark Structured Streaming application that ingested clickstream events into Iceberg once per minute.

Each micro-batch committed new data quickly. That is useful for ingestion latency, but it also created a small data file. We deliberately published five events in separate micro-batches, so the January 9 partition now contains five tiny files.

This is a small example of a common streaming problem.

Frequent ingestion favors frequent commits. Analytical queries, on the other hand, usually work better with fewer, larger files.

Small files do not make the table incorrect. The five events are still valid rows. They simply make the table less efficient to plan and read as their number grows.

To manage this, data platforms periodically combine small files into larger files. This operation is called compaction, and the Spark application that performs it is a compaction job.

In this final part, let’s inspect the files created by our stream, compact them with Iceberg, and verify that the physical files changed while the logical rows stayed the same.

Why Small Files Hurt Performance

Iceberg uses table metadata to discover data files. It does not need to list every object in the table’s S3 directory before planning a query.

Still, every data file must be represented in that metadata. Spark must plan work for the selected files, open them, and read their Parquet metadata before processing their rows.

For five files, the difference is not exciting. For tens of thousands of tiny files, that overhead becomes much more noticeable.

Imagine that two tables contain the same amount of data. One stores it in 100 reasonably sized files, while the other stores it in 100,000 tiny files. The second table has much more metadata to inspect and many more files to open, even though the number of useful bytes is the same.

This is why the best write pattern and the best read layout can pull in different directions:

Streaming ingestion writes frequently so new data becomes available quickly.
Analytical queries prefer fewer files so planning and reading require less overhead.

Compaction connects those two needs. We keep ingesting in small increments, then periodically reorganize the resulting files for efficient reads.

Stopping the Streaming Exercise

Before inspecting the table, stop the Part 15 streaming application with Control-C.

This is not a requirement of Iceberg compaction, but it gives our hands-on exercise a stable before-and-after state. Otherwise, the stream could add a sixth file while we are proudly counting five.

In production, ingestion and maintenance may run concurrently. Teams often compact completed or sufficiently old partitions and use scheduling and retry policies to handle commit conflicts. We will return to that practical choice later.

Inspecting the Current Data Files

Iceberg exposes metadata tables that we can query with Spark SQL.

The files metadata table contains the files referenced by the table’s current snapshot. Let’s inspect the January 9 data files:

python

spark.sql("""
    SELECT
      file_path,
      record_count,
      file_size_in_bytes,
      partition
    FROM oleander.tutorial.retail_clickstream_events.files
    WHERE content = 0
      AND partition.event_time_day = DATE '2026-01-09'
    ORDER BY file_path
""").show(truncate=False)

content = 0 selects data files. Iceberg metadata tables can also describe delete files, but our streaming app only appended data files.

The partition column is a structure containing the table’s partition values. In Part 13, we added day(event_time), and Iceberg named that hidden partition field event_time_day.

We can summarize the same files with another query:

python

spark.sql("""
    SELECT
      COUNT(*) AS data_file_count,
      SUM(record_count) AS record_count,
      SUM(file_size_in_bytes) AS total_file_size_in_bytes
    FROM oleander.tutorial.retail_clickstream_events.files
    WHERE content = 0
      AND partition.event_time_day = DATE '2026-01-09'
""").show(truncate=False)

If you followed Part 15 by publishing one event per micro-batch, this query should report five data files containing five records.

The exact byte count depends on the Parquet files written in your environment. It is likely to look rather extravagant for five rows. Each file needs its own structure and metadata, no matter how lonely its event happens to be.

If you published several events before the next streaming trigger, Spark may have processed them in the same micro-batch. In that case, you may already have fewer than five files. That is fine; the important point is that repeated micro-batches can accumulate small files over time.

What Compaction Changes

Compaction reads eligible data files and writes their rows into a smaller number of output files.

It changes the table’s physical layout, but it does not change its logical contents. After compaction, we expect the same:

schema;
row values;
row count;
daily partition membership.

Iceberg does not combine files across partition boundaries. Our January 9 events remain in the January 9 partition rather than sharing a file with January 8 or January 10.

The rewrite commits atomically and creates a new Iceberg snapshot. Readers see either the snapshot before compaction or the snapshot after compaction, not a half-rewritten table.

Older snapshots may still reference the replaced files. Iceberg therefore cannot immediately delete every old file after the rewrite. Compaction improves the current table layout, but it does not necessarily reclaim the same amount of storage immediately.

Compaction also does not deduplicate rows. If the same event_id was appended twice, those two logical rows remain after the files are combined. Changing file containers is not the same as changing their contents.

Returning to `rewrite_data_files`

We first met Iceberg’s rewrite_data_files procedure in Part 13.

At that time, we had just added partitioning to existing tables. We used rewrite-all=true to reorganize every historical file under the current partition specification.

This time, the purpose is different. We want a routine maintenance job that selects eligible small files from one requested interval. We do not need to force a rewrite of the entire table each time.

Iceberg provides this operation through the rewrite_data_files stored procedure. The official Iceberg Spark procedures documentation lists its strategies, options, and result columns.

Preparing the Compaction Job

Create the Part 16 project with the same Spark template used for our earlier batch jobs:

bash

oleander spark init part16-compaction
cd part16-compaction

Use tutorial_part16_compaction.py as the entrypoint name and tutorial as the Python package name.

Then install the local development environment:

bash

uv sync

Unlike the Part 15 application, this job does not need to reach the Kafka broker on your machine. It reads and rewrites only the Iceberg table, so it can run as a finite batch job on Oleander-managed Spark.

We will reuse the required --data-interval-start and --data-interval-end arguments from the earlier batch jobs. The start is included, the end is excluded, and both values represent UTC boundaries.

Executing Compaction with Spark

The compaction function is short because Iceberg handles the rewrite itself:

python

def compact_clickstream_events(
    spark: SparkSession,
    data_interval_start: str,
    data_interval_end: str,
) -> None:
    """Compact eligible clickstream data files for the given interval."""
    spark.sql(f"""
        CALL oleander.system.rewrite_data_files(
          table => 'tutorial.retail_clickstream_events',
          strategy => 'binpack',
          options => map('min-input-files', '2'),
          where => 'event_time >= TIMESTAMP "{data_interval_start}"
                    AND event_time < TIMESTAMP "{data_interval_end}"'
        )
    """).show(truncate=False)

Let’s look at each part.

Calling an Iceberg Procedure

The procedure name is:

sql

oleander.system.rewrite_data_files

oleander is the Iceberg catalog, and system is the namespace containing Iceberg’s stored procedures.

Because the catalog is already identified there, the table argument uses the namespace and table name without repeating oleander:

sql

table => 'tutorial.retail_clickstream_events'

Using Bin Packing

The strategy is binpack:

sql

strategy => 'binpack'

Bin packing groups eligible files by size and rewrites them into fewer files. It is the default rewrite strategy, but stating it explicitly makes the job’s purpose clear.

Iceberg also supports sort-based rewrite strategies. Those are useful when data ordering matters, but our goal here is simply to combine small files.

Selecting Eligible File Groups

We set the minimum number of input files to two:

sql

options => map('min-input-files', '2')

Iceberg’s default is five. We use two so this small tutorial exercise remains eligible even if a reader produced fewer files than expected.

The procedure still needs at least two eligible files in a file group. After our five files become one, there is nothing useful to combine on the next run.

We do not set rewrite-all. Files that do not meet the rewrite criteria remain untouched.

We also do not set target-file-size-bytes. Iceberg uses the target size configured in the table properties, whose default is 512 MiB. That value is a target rather than a promise: five tiny event files do not contain enough data to produce a 512 MiB output file, so they will become one small—but less tiny—file.

Limiting the Interval

The where argument limits file selection to the requested half-open interval:

sql

where => 'event_time >= TIMESTAMP "{data_interval_start}"
          AND event_time < TIMESTAMP "{data_interval_end}"'

For our exercise, that means January 9 is included and January 10 is excluded.

The predicate selects files that may contain matching rows. It is not a row-level DELETE or WHERE filter that removes other rows from a selected file. When Iceberg rewrites a selected file, every row in that file is preserved.

Our day(event_time) partitioning makes this interval especially useful. Iceberg can project the event-time predicate onto its daily partition values and avoid selecting unrelated days.

Reading the Procedure Result

The procedure returns a result DataFrame, and .show(truncate=False) prints it.

Its columns report:

the number of rewritten data files;
the number of newly added data files;
the number of rewritten bytes;
failed data files;
removed delete files.

For the intended exercise, the important counts are five rewritten data files, one added data file, and zero failures. The byte count depends on your files.

Running the Job

The argument parser is the same pattern used by our earlier batch jobs:

python

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument("--data-interval-start", required=True)
    parser.add_argument("--data-interval-end", required=True)
    return parser.parse_args()

The main() function creates the Spark session, uses UTC, and runs the compaction:

python

def main() -> None:
    args = parse_args()
    spark = (
        SparkSession.builder
        .appName("tutorial-part16-compaction")
        .getOrCreate()
    )

    try:
        spark.conf.set("spark.sql.session.timeZone", "UTC")
        compact_clickstream_events(
            spark=spark,
            data_interval_start=args.data_interval_start,
            data_interval_end=args.data_interval_end,
        )
    finally:
        spark.stop()

The finally block stops Spark whether compaction succeeds or raises an error.

Build the project:

bash

make

Upload the entrypoint:

bash

oleander spark jobs upload tutorial_part16_compaction.py

Then compact the January 9 interval:

bash

oleander spark jobs submit tutorial_part16_compaction.py \
  --namespace tutorial \
  --name tutorial_part16_compaction \
  --args=--data-interval-start=2026-01-09T00:00:00Z \
  --args=--data-interval-end=2026-01-10T00:00:00Z \
  --wait

This job terminates after the procedure finishes. It is maintenance work, not another long-running stream.

Verifying the Result

Return to the PySpark shell. If you kept the same shell open while the external compaction job ran, refresh the table so the session sees the newly committed snapshot:

python

spark.catalog.refreshTable(
    "oleander.tutorial.retail_clickstream_events"
)

This refresh is useful because this shell queried the table before another Spark application changed it. It is different from Part 15’s verification, where we opened a new Spark session with no previously loaded table state.

Now run the file-summary query again:

python

spark.sql("""
    SELECT
      COUNT(*) AS data_file_count,
      SUM(record_count) AS record_count,
      SUM(file_size_in_bytes) AS total_file_size_in_bytes
    FROM oleander.tutorial.retail_clickstream_events.files
    WHERE content = 0
      AND partition.event_time_day = DATE '2026-01-09'
""").show(truncate=False)

The January 9 partition should now contain one data file and the same five records.

Let’s also verify the logical rows directly:

python

spark.sql("""
    SELECT
      event_id,
      event_type,
      event_time,
      ingestion_time
    FROM oleander.tutorial.retail_clickstream_events
    WHERE event_id IN (
      'evt_stream_000001',
      'evt_stream_000002',
      'evt_stream_000003',
      'evt_stream_000004',
      'evt_stream_000005'
    )
    ORDER BY event_id
""").show(truncate=False)

All five events should still be present with the same values.

If you submit the compaction job for the same interval again, it should report zero rewritten and zero added files. The partition now has only one eligible file, so there is no file group to combine.

This makes repeated periodic runs harmless when no compaction is needed. It is not the same kind of transactional retry guarantee introduced with MERGE INTO in Part 12; the maintenance procedure simply finds no eligible work.

Choosing When to Compact

Compaction improves future reads, but the rewrite itself consumes Spark compute and reads and writes data.

Running compaction after every streaming micro-batch would be an impressive way to turn one maintenance problem into another.

Instead, production systems usually run compaction periodically. The schedule may consider:

file count within a partition;
average file size;
how quickly new data arrives;
how soon the partition is queried;
how much ingestion delay is acceptable before maintenance.

There is no universal threshold. A busy clickstream table and a slowly growing reference table should not be expected to use the same schedule.

It is often simpler to compact completed or older partitions rather than the partition receiving the newest events. If compaction and ingestion update the same table concurrently, their Iceberg commits may conflict. Production orchestration should schedule the work carefully and retry appropriate commit failures.

Other Iceberg Maintenance

Data-file compaction is only one kind of Iceberg maintenance.

Long-running tables may also need:

snapshot expiration to remove old snapshots that are no longer within the retention policy;
orphan-file cleanup to find unreferenced files left outside the table’s valid metadata history;
manifest rewriting to reorganize metadata used during scan planning;
delete-file maintenance for tables that accumulate position or equality delete files.

These operations solve different problems and have their own retention and safety considerations. We will not execute them in this beginner exercise.

The important distinction is that compacting current data files does not remove files still referenced by historical snapshots. Snapshot retention and physical cleanup must be designed separately.

Full Code

The complete companion entrypoint is available here:

Part 16 compaction job

Summary

We started this series with the basic building blocks of a data system: compute, storage, tables, files, and catalogs. From there, we introduced Apache Iceberg as the table format and Apache Spark as the engine that reads, transforms, and writes our data.

We learned how Iceberg represents tables through schemas, partitions, snapshots, manifests, and data files. We created Iceberg tables with Spark, loaded our Oleander retail clickstream data, and queried it with both Spark SQL and the DataFrame API.

Then we turned those raw events into useful analytical datasets. We built daily traffic summaries, session-level records, product funnels, and cart-abandonment metrics. Along the way, we used filtering, grouping, aggregations, common table expressions, and joins, and we compared how the same ideas are expressed in SQL and DataFrames.

We also moved beyond transformations and looked at how a pipeline behaves over time. MERGE INTO made interval processing safer to retry. Partitioning helped Spark avoid irrelevant files. Time travel let us inspect historical snapshots before and after late data arrived.

Finally, we replaced our original batch-file ingestion with Kafka and Spark Structured Streaming. We parsed and validated events, used checkpoints to track progress, wrote micro-batches into Iceberg, and compacted the resulting small files for more efficient reads.

These tutorials do not turn our example into a complete production platform. Scheduling, monitoring, access control, schema governance, recovery procedures, and workload-specific tuning still require careful design. They should, however, give us a practical foundation for understanding where those concerns fit.

A healthy Iceberg and Spark pipeline is more than a query or a table. It includes ingestion, transformation, reliable writes, useful physical design, historical visibility, and ongoing maintenance.

That brings our Iceberg and Spark 101 journey to an end. The table, fortunately, will keep going.

PreviousStreaming Clickstream Events into Iceberg The endYour garden is plantedBack to the guide

Managing Small Files with Compaction

Why Small Files Hurt Performance

Stopping the Streaming Exercise

Inspecting the Current Data Files

What Compaction Changes

Returning to rewrite_data_files

Preparing the Compaction Job

Executing Compaction with Spark

Calling an Iceberg Procedure

Using Bin Packing

Selecting Eligible File Groups

Limiting the Interval

Reading the Procedure Result

Running the Job

Verifying the Result

Choosing When to Compact

Other Iceberg Maintenance

Full Code

Summary

Returning to `rewrite_data_files`