#Parquet Page Index and Bloom Filters for Spatial Queries

Row-group statistics get a reader down to about 128 MB of candidate data; the page index gets it down to about a megabyte, and a bloom filter answers point lookups without reading data at all. Both structures are cheap to write, both live in the footer where a reader can fetch them in one request, and both are ignored by default in enough engines that they are worth verifying rather than assuming. This page continues understanding Parquet columnar storage for GIS, one level finer than bbox covering columns.

#Quick Reference

#Three Levels of “Do Not Read This”

Parquet’s skipping story is a hierarchy, and each level narrows what the level above admitted.

At the top, row-group statistics in the footer give min and max per column per row group. With covering bbox columns present these become a spatial filter, and they remove most of the file for a windowed query. What they cannot do is help inside a surviving group: once the engine decides to read a 128 MB row group, it reads all of it for the columns it needs.

The page index closes that gap. A column chunk is physically a sequence of pages, each typically around a megabyte, and the page index records min, max, and null count for every one of them — stored in the footer, not inline. That location is the entire point: a reader fetches the index in one request and then issues reads only for the pages whose statistics survive the predicate. Without the separation it would have to read pages to discover whether to read them.

Bloom filters answer a different question. They are compact probabilistic structures that can say definitely not present for an exact value, with a small false-positive rate and no false negatives. For a query like WHERE uprn = '100023336956' across a dataset partitioned by region, a bloom filter per column chunk eliminates almost every chunk without touching the data, turning a full scan into a handful of reads.

#Writing Them

# Requires: pyarrow>=16, geopandas>=1.0  (Python 3.10+)
from __future__ import annotations

import geopandas as gpd
import pyarrow as pa
import pyarrow.parquet as pq


def write_with_fine_grained_index(
    table: pa.Table,
    target: str,
    *,
    bloom_columns: tuple[str, ...] = (),
    row_group_rows: int = 200_000,
    page_size_bytes: int = 1 << 20,
) -> dict[str, object]:
    """Write GeoParquet with page-level statistics and targeted bloom filters."""
    missing = [c for c in bloom_columns if c not in table.column_names]
    if missing:
        raise KeyError(f"bloom filter requested for absent columns: {missing}")

    for column in bloom_columns:
        # A bloom filter over a low-cardinality column is wasted space: the
        # dictionary page already answers the question more precisely.
        distinct = pa.compute.count_distinct(table[column]).as_py()
        if distinct < 1000:
            raise ValueError(
                f"{column} has only {distinct} distinct values — use the "
                f"dictionary page rather than a bloom filter"
            )

    pq.write_table(
        table, target,
        row_group_size=row_group_rows,
        data_page_size=page_size_bytes,
        compression="zstd",
        compression_level=3,
        write_statistics=True,
        # The page index is the footer-resident per-page min/max structure.
        write_page_index=True,
        write_bloom_filter=list(bloom_columns) if bloom_columns else False,
    )

    meta = pq.read_metadata(target)
    return {
        "row_groups": meta.num_row_groups,
        "rows": meta.num_rows,
        "page_index": bool(meta.row_group(0).column(0).offset_index_offset),
        "bloom_columns": list(bloom_columns),
    }

#Validation

The check is empirical: run a selective query with the structures present and confirm bytes read collapses. Metadata presence alone proves nothing, because the reader may ignore it.

# Requires: pyarrow>=16 — confirm the page index is present and populated
import pyarrow.parquet as pq

meta = pq.read_metadata("parcels.parquet")
group = meta.row_group(0)
column = group.column(0)
assert column.offset_index_offset, "no page index was written"
assert column.column_index_offset, "no page-level statistics were written"
print(f"{meta.num_row_groups} row groups, page index present, "
      f"{group.total_byte_size / 1e6:.0f} MB per group")

-- DuckDB: the same selective query, measured. Compare against a file written
-- without write_page_index to see whether the reader is using it.
SET enable_profiling = 'json';
SELECT count(*) FROM read_parquet('parcels.parquet')
WHERE bbox.xmin <= -2.44 AND bbox.xmax >= -2.46
  AND bbox.ymin <= 51.46 AND bbox.ymax >= 51.44;

Healthy result: bytes read for a very selective window should be single-digit megabytes on a multi-gigabyte file. If it lands in the hundreds of megabytes, the reader is pruning row groups but not pages — either the index was not written or the engine is not consulting it.

#Edge Cases and Caveats

Page size set too large. A 32 MB page is a second row group with none of the benefits: the index gets coarse and the skipping stops paying. Keep pages around a megabyte so a page is a genuinely small unit of work, and let row group size carry the coarse-grained trade described in row group sizing strategies.

Bloom filters on low-cardinality columns. A filter over a column with forty distinct values is strictly worse than the dictionary page that already exists for it — it costs space and answers less precisely. Reserve bloom filters for identifiers and other high-cardinality lookup keys, which is why the implementation above refuses to write one below a thousand distinct values.

Page-level statistics on unsorted columns. Like row-group statistics, page statistics only skip when values within a page are clustered. On an unsorted column every page spans the full value range and the index is dead weight. The sort that helps here is the same space-filling-curve ordering that makes bbox statistics useful.

#Frequently Asked Questions

What does the Parquet page index add over row-group statistics?

Resolution. Row-group statistics let an engine skip a whole row group, which is typically 128 MB; page-level statistics let it skip individual pages inside a group, which are typically a megabyte or less. The important structural detail is that the page index lives in the footer rather than inline with the pages, so a reader fetches all the page statistics in one request and then reads only the pages that survive — without that separation it would have to read the pages to learn whether to read them.

Are bloom filters useful for spatial data?

Not for geometry, but very much for the identifier columns beside it. A bloom filter answers ‘is this value definitely absent?’ for equality predicates, which is exactly the shape of a feature-by-identifier lookup across a partitioned dataset. It cannot help a range or intersection predicate, so it complements bbox covering columns rather than competing with them: the covering handles where, the bloom filter handles which.

Why does my engine ignore the page index I wrote?

Support is uneven and frequently opt-in. Some readers only consult the page index when a specific flag is set, some use it for numeric columns but not nested struct fields, and some ignore bloom filters entirely. Writing them costs little and is harmless when unused, but do not assume a benefit you have not measured — check bytes read for a selective query with and without the structures present.

#Related

• Understanding Parquet Columnar Storage for GIS — parent guide: the physical layout these structures describe
• GeoParquet bbox Covering Columns Explained — the coarse spatial filter this one refines
• Column Pruning Benefits in Geospatial Parquet — narrowing by column rather than by row
• Dictionary Encoding for Categorical GIS Attributes — the structure that replaces a bloom filter on low-cardinality columns

← Back to Understanding Parquet Columnar Storage for GIS