#Column Pruning Benefits in Geospatial Parquet

Column pruning eliminates unnecessary I/O by reading only the columns a query explicitly requests, bypassing heavy geometry, raster, or attribute fields entirely. In GeoParquet files, this yields 60–90% faster execution for attribute-only queries on parcels or land-use datasets, sharply reduced cloud egress costs, and lower memory pressure during spatial joins and aggregations. The optimization works because Parquet’s columnar layout stores each field independently on disk: a query engine consults file metadata, maps requested columns to byte offsets, and issues targeted range-reads against object storage before any deserialization occurs.

#Quick-Reference: What to Prune and When

The largest gains occur whenever geometry — often 70–85% of a GeoParquet file’s footprint when encoded as WKB — is not required by the query.

#Why Column Pruning Works: Parquet’s Physical Layout

Parquet files embed a footer containing column-level statistics (min/max values, null counts, byte offsets) and row-group boundaries. When a query engine executes SELECT region_id, land_use_class FROM parcels, it reads the footer first, maps the two requested columns to their physical byte ranges, then issues precisely those HTTP range-reads against S3, GCS, or Azure Blob. The geometry column’s pages are never fetched.

This projection pushdown is a central mechanism in columnar Parquet storage for GIS: query planners delegate filtering and projection to the storage layer rather than pulling full rows into memory. When geometry is not requested, the binary decoding step — WKB deserialisation is CPU-intensive and causes large heap allocations — is skipped entirely, cutting both latency and memory pressure.

Pairing column pruning with row group sizing amplifies the effect: larger row groups mean fewer footer round-trips; smaller row groups make row-group skipping more selective. The two knobs interact, so benchmark both together. For geometry columns that must be loaded, ZSTD compression at levels 1–3 reduces transfer size without adding meaningful decode latency.

#Production Implementation

The PyArrow dataset API is preferred over the legacy pq.read_table for cloud-native workflows because it supports partition discovery and pushes the projection down to the file reader rather than loading then slicing.

# pyarrow>=12.0, boto3 configured in environment
import pyarrow.dataset as ds
import pyarrow.fs as fs
from typing import Optional

# Columns to project — geometry intentionally excluded
PROJECTED_COLUMNS: list[str] = [
    "parcel_id",
    "zoning_code",
    "assessed_value",
    "last_updated",
]


def read_parcels_attributes(
    s3_prefix: str,
    columns: Optional[list[str]] = None,
    value_floor: Optional[int] = None,
) -> "pyarrow.Table":
    """
    Return only the requested attribute columns from a partitioned
    GeoParquet dataset. Geometry is excluded unless explicitly requested.

    Args:
        s3_prefix:   S3 path, e.g. 's3://gis-lake/parcels/year=2024/'
        columns:     list of column names to project; defaults to PROJECTED_COLUMNS
        value_floor: optional predicate on assessed_value (pushed to reader)

    Returns:
        pyarrow.Table with only the projected columns loaded
    """
    if columns is None:
        columns = PROJECTED_COLUMNS

    s3 = fs.S3FileSystem()

    dataset = ds.dataset(
        s3_prefix.removeprefix("s3://"),
        filesystem=s3,
        format="parquet",
        partitioning="hive",  # recognises year= / month= directories
    )

    # Build optional predicate — pushed down to row-group reader
    filt = None
    if value_floor is not None:
        filt = ds.field("assessed_value") > value_floor

    table = dataset.to_table(
        columns=columns,
        filter=filt,
        use_threads=True,
    )
    print(
        f"Loaded {table.num_rows:,} rows | "
        f"{len(columns)} columns | "
        f"~{table.nbytes / 1_048_576:.1f} MB in memory"
    )
    return table


if __name__ == "__main__":
    result = read_parcels_attributes(
        "s3://gis-lake/parcels/year=2024/month=11/",
        value_floor=500_000,
    )

When geometry is needed later, add "geometry" to the columns list rather than loading it by default.

#Validation and Verification

Confirm that pruning is active before committing a pattern to production. DuckDB’s EXPLAIN output is the quickest check:

-- DuckDB 1.0+; install spatial extension only if ST_* functions are needed
EXPLAIN
SELECT parcel_id, zoning_code, assessed_value
FROM read_parquet('s3://gis-lake/parcels/*.parquet')
WHERE assessed_value > 500000;

In the output, look for a PROJECTION node listing only the three requested columns. If geometry appears in the scan node despite not being in the SELECT list, projection pushdown is not firing — usually because the file lacks column statistics or the reader version predates pushdown support.

For PyArrow, measure I/O reduction empirically:

# pyarrow>=12.0
import pyarrow.parquet as pq
import pyarrow.dataset as ds

local_file = "parcels_sample.parquet"
meta = pq.read_metadata(local_file)

# Report bytes per column chunk in row group 0
for i in range(meta.row_group(0).num_columns):
    col = meta.row_group(0).column(i)
    print(f"{col.path_in_schema}: {col.total_compressed_size:,} bytes")

# Compare memory footprint with and without geometry
full   = pq.read_table(local_file)
pruned = pq.read_table(local_file, columns=PROJECTED_COLUMNS)
saving_pct = 100 * (1 - pruned.nbytes / full.nbytes)
print(f"Memory saving from pruning: {saving_pct:.0f}%")

Expected output ranges: attribute-only reads consume 10–30% of the bytes used by a full row that includes WKB geometry. A result above 50% savings confirms geometry was previously being loaded unintentionally — exactly the kind of silent waste that accumulates across long-running pipelines.

#Edge Cases and Caveats

When geometry is unavoidably required. Some operational datasets store geometry as uncompressed WKB alongside dozens of numeric attributes. Even with column pruning, if the downstream step requires ST_Intersects, the geometry column must be loaded. In these cases, pair pruning with ZSTD compression on the geometry column at a low level (1–3) to reduce transfer size without adding decode latency. See spatial partitioning with quadtree indexes for strategies that reduce how many geometry pages need loading in the first place.

Schema evolution across file vintages. A partitioned dataset spanning multiple years may have different column schemas in older vs. newer files. Requesting a column that does not exist in older Parquet files causes pyarrow to raise a KeyError unless you pass schema= explicitly or set ignore_prefixes in the partition config. Always test pruned reads against the oldest partition in your dataset.

Mixed CRS sources and bounding-box columns. If bbox_min_x / bbox_max_x columns were generated in EPSG:4326 but the dataset has since been re-projected, numeric bounding-box filters return incorrect spatial results. Verify that the CRS metadata in the GeoParquet file’s geo metadata key matches the bbox column generation CRS before relying on pre-computed extents as a pruning proxy — see the CRS handling notes in GeoParquet vs. FlatGeobuf performance.

Non-pushdown filter patterns to avoid.

• df[df['value'] > 100] applied after pq.read_table() loads the full table first, then filters in memory — no pruning occurs.
• SELECT * in DuckDB reads all column pages; always enumerate the columns you need.
• Writing Parquet with write_statistics=False disables row-group skipping on predicate columns; always enable statistics for production datasets.

#Frequently Asked Questions

Does column pruning work automatically in DuckDB and PyArrow?

DuckDB pushes projections down to the Parquet reader automatically when you issue a SELECT listing specific columns — no extra configuration is required. PyArrow’s dataset API requires you to pass an explicit columns= list; the legacy pq.read_table function accepts columns= too but does not support partition discovery. Always check with EXPLAIN that the geometry column is absent from the scan node before treating the optimisation as confirmed.

How much I/O does skipping the geometry column actually save?

Geometry encoded as WKB typically occupies 70–85% of a GeoParquet file’s compressed footprint. Attribute-only queries that skip this column cut disk reads and object-storage GET payload by the same proportion — commonly 60–90% fewer bytes transferred per query. On a 10 GB parcels file, that translates directly to reduced S3 egress charges and sub-second query latency instead of tens of seconds.

Can I prune bounding-box columns instead of computing them from WKB?

Yes. Storing bbox_min_x, bbox_max_x, bbox_min_y, bbox_max_y as separate numeric columns lets you run bounding-box filters as pure numeric comparisons without touching the binary geometry field at all. The GeoParquet spec’s bbox metadata key exists for exactly this reason. Combine this with row group sizing tuned to your tile dimensions and you can answer viewport-bounded queries with a small fraction of total file I/O.

What breaks column pruning in practice?

Four common failure modes: (1) the Parquet writer has write_statistics=False so row-group skipping cannot occur even when predicates are pushed down; (2) deeply nested struct columns force partial deserialization of parent fields; (3) Python-side DataFrame filters like df[df['col'] > x] are applied after loading the full table rather than pushed to the reader; and (4) schema evolution leaves columns missing in older file partitions without explicit schema-merging configuration, causing silent fallback to full-table reads.

#Related

• Understanding Parquet Columnar Storage for GIS — architecture of row groups, column chunks, and dictionary encoding for geospatial workloads
• Row Group Sizing Strategies for Parquet — how row group size interacts with column pruning efficiency
• ZSTD Compression Levels for Geospatial Data — compression choices for geometry columns that cannot be pruned
• GeoParquet vs. FlatGeobuf Performance — when columnar pruning advantages outweigh sequential-read formats
• Spatial Partitioning with Quadtree Indexes — partitioning strategies that amplify column pruning gains

← Back to Understanding Parquet Columnar Storage for GIS