Data Compression
  1. Home
  2. Query Engines & Cloud Analytics for GeoParquet
  3. Trino & Presto GeoParquet Connector
  4. Iceberg Tables with GeoParquet for a Spatial Lakehouse
Section: Query Engines & Cloud Analytics for GeoParquet 13 min read

Iceberg Tables with GeoParquet for a Spatial Lakehouse

Iceberg does not make GeoParquet faster; it makes it transactional. The space-filling-curve sort, the covering bbox columns, and the row-group statistics that make spatial queries cheap are all properties of the files, and they are unchanged. What changes is that the set of files becomes a table with atomic commits, snapshot isolation, time travel, and schema evolution — which is exactly the machinery a hand-rolled manifest approximates. This page extends the Trino and Presto GeoParquet connector.

Quick Reference

Concern Handled by Notes Primary Use Case
Spatial skipping GeoParquet Covering bbox + Hilbert sort Every spatial query
Column pruning GeoParquet Unchanged by the table format Wide attribute schemas
Atomic commit Iceberg Replaces a manual pointer swap Multi-writer pipelines
Snapshot isolation Iceberg Readers see a consistent file list Long analytical queries
Partition pruning Iceberg On a materialised spatial key Region-scoped queries
Schema evolution Iceberg Rename and add without rewriting Long-lived tables
Small-file compaction You, on a schedule Neither format does it automatically Incremental pipelines

What Belongs at Which Layer

The clearest way to reason about a spatial lakehouse is to keep the two layers separate in your head, because almost every confusion comes from expecting one to do the other’s job.

The file layer is GeoParquet, and it owns everything spatial. Rows sorted on a Hilbert curve so that spatially close features are physically close; covering bbox columns so a row group can be ruled out from the footer; row groups sized so skipping has useful granularity. Take Iceberg away and every one of these still works.

The table layer is Iceberg, and it owns everything transactional. A snapshot is an immutable list of data files with their statistics; a commit atomically replaces one snapshot with another; a reader pins a snapshot for the life of its query. Take the geometry away and every one of these still works.

The one place they meet is partitioning. Iceberg prunes on partition values, and it can only prune on columns that exist. Materialising a spatial key — a Hilbert bucket, an H3 cell, a grid identifier — as a real column turns Iceberg’s ordinary partition pruning into spatial pruning, without teaching it anything about geometry.

Which layer owns which capability Two stacked layers. The upper Iceberg layer owns snapshots, atomic commits, schema evolution, and partition pruning on declared columns, and treats geometry as an opaque binary column. The lower GeoParquet layer owns the Hilbert sort, covering bounding-box columns, row-group statistics, and column pruning. A single bridge between them is a materialised spatial key column, which lets the table layer prune spatially without understanding geometry. A note records that removing either layer leaves the other fully functional. Iceberg — the table layer, and it owns nothing spatial snapshots immutable file lists atomic commit multi-writer safe schema evolution rename without rewrite partition pruning on declared columns materialised spatial key column the one bridge between the layers GeoParquet — the file layer, and it owns everything spatial Hilbert sort covering bbox columns row-group statistics column pruning Remove either layer and the other keeps working — which is why they compose cleanly.

Creating the Table

python
# Requires: pyiceberg>=0.7, pyarrow>=16, geopandas>=1.0  (Python 3.10+)
from __future__ import annotations

import numpy as np
import geopandas as gpd
import pyarrow as pa
from pyiceberg.catalog import load_catalog
from pyiceberg.exceptions import NoSuchTableError
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.transforms import IdentityTransform

HILBERT_ORDER = 12          # 4096 × 4096 grid over the CRS extent


def add_spatial_key(frame: gpd.GeoDataFrame, *, buckets: int = 4096) -> gpd.GeoDataFrame:
    """Materialise the spatial partition key Iceberg will prune on.

    Iceberg cannot derive this itself — it has no notion of geometry — so the
    key has to be an ordinary column before the table is created.
    """
    if frame.crs is None:
        raise ValueError("a CRS is required before deriving a spatial key")
    distance = frame.geometry.hilbert_distance(level=HILBERT_ORDER)
    out = frame.copy()
    # Bucketing the Hilbert index keeps the partition count bounded while
    # preserving locality: adjacent buckets are adjacent in space.
    out["hilbert_bucket"] = (distance // max(1, (4 ** HILBERT_ORDER) // buckets)).astype("int32")
    return out.sort_values("hilbert_bucket").reset_index(drop=True)


def create_table(catalog_name: str, identifier: str, frame: gpd.GeoDataFrame):
    """Create an Iceberg table over spatially laid-out GeoParquet."""
    catalog = load_catalog(catalog_name)
    prepared = add_spatial_key(frame)

    table_arrow = pa.Table.from_pandas(
        prepared.drop(columns=prepared.geometry.name), preserve_index=False
    ).append_column(
        "geometry", pa.array(prepared.geometry.to_wkb(), type=pa.binary())
    )

    try:
        catalog.drop_table(identifier)
    except NoSuchTableError:
        pass

    spec = PartitionSpec(
        PartitionField(
            source_id=table_arrow.schema.get_field_index("hilbert_bucket") + 1,
            field_id=1000,
            transform=IdentityTransform(),
            name="hilbert_bucket",
        )
    )
    table = catalog.create_table(identifier, schema=table_arrow.schema, partition_spec=spec)
    table.append(table_arrow)
    return table


def commit_incremental(table, replacement: pa.Table, buckets: set[int]) -> None:
    """Replace whole spatial partitions in one transaction.

    This is the operation a hand-rolled manifest and pointer swap approximates;
    here the format guarantees readers never see a partial state.
    """
    if not buckets:
        raise ValueError("no partitions to replace")
    predicate = " OR ".join(f"hilbert_bucket = {b}" for b in sorted(buckets))
    with table.transaction() as txn:
        txn.overwrite(replacement, overwrite_filter=predicate)

Validation

Confirm two things: that partition pruning fires on the spatial key, and that row-group skipping still fires inside the surviving files. Both must work; either alone leaves most of the benefit on the table.

sql
-- Trino: partition pruning at the table layer, row-group skipping at the file layer
EXPLAIN ANALYZE
SELECT count(*) FROM iceberg.gis.parcels
WHERE hilbert_bucket BETWEEN 1840 AND 1856
  AND bbox.xmin <= -2.44 AND bbox.xmax >= -2.46
  AND bbox.ymin <= 51.46 AND bbox.ymax >= 51.44;

-- Look for both:
--   "input: 17 files"        ← Iceberg pruned 4,283 of 4,300 partitions
--   "physical input: 41 MB"  ← Parquet skipped row groups inside those 17 files

Expected results: file count reduced by two to three orders of magnitude by partition pruning, and physical bytes read a small fraction of the surviving files’ total size. If files are pruned but physical input approaches their full size, the covering columns or the sort are missing and the file layer is doing nothing.

Both layers must prune, and they prune in sequence A funnel with three stages for one regional query. The table starts with four thousand three hundred partitions and one point two terabytes. Iceberg's partition pruning on the materialised Hilbert bucket reduces this to seventeen files and about four gigabytes. Parquet's row-group skipping using the covering bounding-box columns then reduces it to forty-one megabytes physically read. A note observes that if only the first stage fires, the query reads four gigabytes instead of forty-one megabytes. One regional query through both layers whole table — 4,300 partitions, 1.2 TB catalog metadata read, no data touched after Iceberg partition pruning — 17 files, 4.1 GB hilbert_bucket predicate, decided from table metadata 41 MB physically read Parquet row-group skipping If only the first stage fires, the query reads 4.1 GB instead of 41 MB — a hundredfold difference that no amount of table-format tuning recovers, because it lives in the files. What retained snapshots cost Storage occupied by a table over six months of daily incremental writes. The live data grows slowly, tracking the real dataset. Retained snapshots pin every superseded file, so total storage grows far faster than the data does. Expiring snapshots older than the rollback window and cleaning orphaned files brings the total back to just above the live size. Storage occupied over six months of daily writes live data — grows with the dataset total with retained snapshots expiry + orphan cleanup runs Time travel is only useful with retained snapshots, and retained snapshots pin the files they reference. Schedule expiry and orphan cleanup as part of the pipeline, not as maintenance somebody remembers.

Edge Cases and Caveats

Expecting the table format to sort the data. Iceberg has a write-order concept, but it will not compute a Hilbert index or write covering columns. If the spatial layout work stops when the table format is adopted, queries get slower — the most common disappointment in a spatial lakehouse migration.

Partition counts chosen from geography rather than from file size. A partition per grid cell over a country produces thousands of partitions holding a few megabytes each, which is the small-file problem with extra metadata. Bucket the spatial key so each partition holds a sensible number of properly-sized files.

Snapshots retained forever. Time travel is only useful with retained snapshots, and retained snapshots pin the data files they reference, so storage grows without bound. Schedule snapshot expiry and orphan-file cleanup, and treat them as part of the pipeline exactly as incremental conversion treats its retention job.

Assuming every engine reads the table identically. Iceberg support varies across Trino, Spark, and DuckDB in which predicates get pushed down and which metadata is used. Verify the plan on each engine you actually query with, rather than on the one used for development.

Frequently Asked Questions

Does Iceberg understand spatial data?

No. It manages a list of data files, their schemas, their partition values, and their column statistics, and it treats a geometry column as an opaque binary column like any other. Every spatial optimisation still comes from the GeoParquet underneath — the space-filling-curve sort, the covering bbox columns, the row-group statistics. What Iceberg adds is transactional metadata over those files, which is orthogonal to whether the data is spatial.

What does a table format add over a manifest file I write myself?

Concurrency control, snapshot isolation, and schema evolution that you would otherwise implement and maintain. A hand-rolled manifest with an atomic pointer swap covers a single-writer pipeline well. The moment two processes write the same table, or readers need a consistent view during a rewrite, or a column needs renaming without rewriting the data, you are reimplementing a table format — usually less completely.

How do I get spatial partition pruning in Iceberg?

Materialise a spatial key as an ordinary column — a Hilbert curve bucket, an H3 cell, or a grid identifier — and declare it as the partition field. Iceberg then prunes on that column like any other, and because the key is derived from geometry, pruning on it is spatial pruning. Hidden partitioning means queries filter on the geometry-derived value without having to name the partition column, so readers never write partition predicates by hand.

← Back to Trino and Presto GeoParquet Connector