#Row Group Sizing Strategies for Parquet

Parquet’s columnar architecture relies on row groups as the fundamental unit of storage, decompression, and I/O scheduling. For geospatial workloads — where coordinate arrays, topology rings, and attribute tables scale non-linearly — improper row group configuration directly raises query latency, memory footprint, and cloud storage costs. This guide provides a production-tested workflow for GIS data engineers, Python backend developers, and cloud architects to calculate, configure, and validate optimal chunk boundaries without sacrificing spatial query performance or breaking downstream indexing pipelines.

This guide is part of the Compression, Chunking & Spatial Indexing engineering guide, where row group sizing interacts with codec selection, dictionary encoding, and spatial partitioning as a coupled system — not a standalone dial. Tune the boundary in isolation and you will trade a memory win for a compression loss, or a scan-cost win for a metadata-overhead regression.

#Prerequisites

• Python 3.9+ with pyarrow >= 12.0, geopandas >= 0.13, and zstandard >= 0.21
• Representative sample data: at least 50 000 features covering the geometry complexity range of your production dataset (simple polygons through multi-part rings)
• Columnar storage familiarity: Parquet metadata structure, column chunks, data pages, and predicate pushdown semantics
• Monitoring tools: psutil for memory tracking, time.perf_counter for latency measurement
• Cloud storage access: an S3-compatible endpoint for validating real GET-request cost under your target row group regime

#Architectural Foundations

A row group contains a contiguous subset of rows across all columns in a Parquet file. Within each row group, individual columns are stored as column chunks, which are further subdivided into data pages. The row group boundary dictates how much data must be fetched from storage, decompressed, and held in memory before a query engine can evaluate filters or aggregations.

When working with spatial datasets, misaligned row groups frequently force engines to scan irrelevant coordinate blocks — inflating cloud query costs and increasing memory pressure. This physical layout is governed by the Apache Parquet specification, which establishes that row groups should be large enough to amortize I/O overhead but small enough to fit comfortably in memory during predicate evaluation.

Understanding Parquet columnar storage for GIS is foundational before tuning group boundaries. Three sibling techniques interact with those boundaries directly: ZSTD compression levels determine how efficiently each column chunk compresses within its boundary; dictionary encoding for categorical GIS attributes depends on the dictionary fitting inside the row group; and spatial partitioning with quadtree indexes determines which rows land in the same group when you sort before writing.

#Core Trade-offs in Geospatial Workloads

The default PyArrow row group size (typically 64 MB–128 MB uncompressed) works adequately for generic tabular data but frequently underperforms for high-precision geometries, dense point clouds, or multi-part polygons. Optimal sizing depends on three interdependent variables:

Query engine I/O patterns. Cloud engines like Athena prefer 128 MB–256 MB row groups to maximise sequential read throughput and amortise S3 GET request overhead. Local or interactive engines (DuckDB, Polars) benefit from smaller 32 MB–64 MB groups for faster predicate evaluation and lower memory pressure.

Geometry complexity. WKB-encoded polygons with thousands of vertices inflate row group sizes unpredictably. A dataset of simple bounding-box polygons might fit 500 000 rows in 64 MB; a dataset of coastline features with the same row count could exceed 2 GB. Aligning boundaries with representative geometry samples prevents partial reads and topology fragmentation in downstream GIS tools.

Compression synergy. Row group size directly influences dictionary reuse, delta encoding effectiveness, and ZSTD’s internal match-finder window. Oversized groups exhaust dictionary memory limits, forcing fallback to plain encoding, while undersized groups fragment statistical distributions and degrade compression ratios by 10–20% on coordinate columns.

These forces pull in opposite directions, which is why query latency traces a U-shaped curve against row group size rather than improving monotonically. Below the sweet spot, per-request I/O overhead and weaker compression dominate; above it, the engine wastes work decompressing and pruning more data than a selective spatial filter actually needs.

Categorical attributes such as land-use codes, administrative boundaries, or sensor types also interact with chunk boundaries. When row groups exceed the cardinality threshold for dictionary encoding, PyArrow falls back to plain encoding automatically — increasing storage footprint and I/O latency without warning. Properly scoping row group size ensures dictionary encoding remains effective across the whole dataset.

#Step-by-Step Workflow

#Step 1 — Profile Your Data’s Row Footprint

Before configuring anything, measure the uncompressed byte cost of an average row. Geometry serialization dominates this figure for vector datasets and varies by order of magnitude between geometry types.

# pyarrow>=12.0, geopandas>=0.13
import geopandas as gpd
import numpy as np


def profile_row_footprint(gdf: gpd.GeoDataFrame, sample_n: int = 5_000) -> dict[str, float]:
    """
    Estimate mean uncompressed bytes per row, split by geometry and attributes.

    Args:
        gdf: GeoDataFrame with representative geometry distribution.
        sample_n: Number of rows to sample. Use >=5 000 for heterogeneous geometry.

    Returns:
        Dictionary with keys 'attr_bytes_per_row', 'geom_bytes_per_row', 'total_bytes_per_row'.
    """
    if len(gdf) < sample_n:
        sample = gdf
    else:
        sample = gdf.sample(n=sample_n, random_state=42)

    # Attribute columns only (exclude geometry)
    attr_cols = [c for c in sample.columns if c != sample.geometry.name]
    attr_df = sample[attr_cols]
    attr_bytes = attr_df.memory_usage(deep=True, index=False).sum()
    attr_per_row = attr_bytes / len(sample)

    # WKB serialization for geometry column
    wkb_sizes = sample.geometry.apply(
        lambda g: len(g.wkb) if g is not None else 0
    )
    geom_per_row = float(wkb_sizes.mean())
    geom_p95 = float(np.percentile(wkb_sizes, 95))

    return {
        "attr_bytes_per_row": attr_per_row,
        "geom_bytes_per_row": geom_per_row,
        "geom_p95_bytes": geom_p95,
        "total_bytes_per_row": attr_per_row + geom_per_row,
    }

Pay attention to the p95 geometry size. If the p95 is more than 3× the mean, the dataset has pathological outlier geometries that will cause individual row groups to blow past your byte target. Clip or simplify those features before writing.

#Step 2 — Map Your Workload to an Engine Target

Convert your profiling output to a row count target using the engine profile that matches your primary query workload:

For detailed trade-off analysis of cloud-specific tuning, the child page Tuning Row Group Size for Cloud Query Performance covers S3 GET-request economics, Athena scan cost modelling, and Spark partition alignment.

#Step 3 — Configure Parameters

PyArrow’s pq.write_table accepts row_group_size as a row count, not bytes. Convert your target using the profiled footprint:

import math


def calculate_row_group_count(
    bytes_per_row: float,
    target_mb: int,
    min_rows: int = 5_000,
    max_rows: int = 2_000_000,
) -> int:
    """
    Convert an uncompressed MB target into a row count for pq.write_table.

    Args:
        bytes_per_row: From profile_row_footprint()['total_bytes_per_row'].
        target_mb: Desired uncompressed size per row group in megabytes.
        min_rows: Floor — prevents excessively small groups on tiny datasets.
        max_rows: Ceiling — prevents runaway groups on complex geometry datasets.

    Returns:
        Row count to pass as row_group_size to pq.write_table.
    """
    if bytes_per_row <= 0:
        raise ValueError(f"bytes_per_row must be positive, got {bytes_per_row}")
    target_bytes = target_mb * 1024 * 1024
    raw_count = int(math.ceil(target_bytes / bytes_per_row))
    return max(min_rows, min(raw_count, max_rows))

#Step 4 — Write with Explicit Sizing

import pyarrow as pa
import pyarrow.parquet as pq
import geopandas as gpd


def write_geoparquet_sized(
    gdf: gpd.GeoDataFrame,
    output_path: str,
    target_mb: int = 128,
    compression_level: int = 3,
) -> None:
    """
    Write a GeoDataFrame to Parquet with calculated row group sizing.

    Args:
        gdf: Input GeoDataFrame. Must be non-empty.
        output_path: Destination file path (local or s3://).
        target_mb: Target uncompressed size per row group in MB.
        compression_level: ZSTD level (1–22; 3 balances ratio and read speed).

    Raises:
        ValueError: If gdf is empty or byte-per-row estimate is zero.
    """
    if gdf.empty:
        raise ValueError("Input GeoDataFrame is empty — nothing to write.")

    footprint = profile_row_footprint(gdf)
    rows_per_group = calculate_row_group_count(
        bytes_per_row=footprint["total_bytes_per_row"],
        target_mb=target_mb,
    )

    table = pa.Table.from_pandas(gdf, preserve_index=False)

    pq.write_table(
        table,
        output_path,
        row_group_size=rows_per_group,
        use_dictionary=True,
        write_statistics=True,
        compression="zstd",
        compression_level=compression_level,
    )

#Step 5 — Benchmark Against Alternatives

Run at least three row group configurations and measure both write-time metadata and query-time latency:

import time
import pyarrow.parquet as pq


def benchmark_row_group_candidates(
    gdf: gpd.GeoDataFrame,
    candidates_mb: list[int],
    tmp_dir: str,
) -> list[dict]:
    """
    Write gdf at multiple row group sizes and measure compression + write time.

    Returns list of result dicts for building a comparison table.
    """
    results = []
    for target_mb in candidates_mb:
        path = f"{tmp_dir}/rg_{target_mb}mb.parquet"
        t0 = time.perf_counter()
        write_geoparquet_sized(gdf, path, target_mb=target_mb)
        elapsed = time.perf_counter() - t0

        meta = pq.read_metadata(path)
        total_compressed = sum(
            meta.row_group(i).total_byte_size
            for i in range(meta.num_row_groups)
        )
        results.append({
            "target_mb": target_mb,
            "num_row_groups": meta.num_row_groups,
            "compressed_mb": round(total_compressed / 1024 / 1024, 1),
            "write_s": round(elapsed, 2),
        })
    return results

#Step 6 — Validate Metadata Post-Write

def validate_parquet_sizing(file_path: str) -> dict:
    """
    Inspect row group boundaries, compressed sizes, and column statistics.

    Returns a summary dict for logging or alerting pipelines.
    """
    meta = pq.read_metadata(file_path)
    rg_sizes = [
        meta.row_group(i).total_byte_size for i in range(meta.num_row_groups)
    ]
    stats_present = sum(
        1
        for i in range(meta.num_row_groups)
        for j in range(meta.row_group(i).num_columns)
        if meta.row_group(i).column(j).is_stats_set
    )
    return {
        "num_row_groups": meta.num_row_groups,
        "total_rows": meta.num_rows,
        "compressed_mb_min": round(min(rg_sizes) / 1024 / 1024, 2),
        "compressed_mb_max": round(max(rg_sizes) / 1024 / 1024, 2),
        "compressed_mb_mean": round(sum(rg_sizes) / len(rg_sizes) / 1024 / 1024, 2),
        "columns_with_statistics": stats_present,
    }

Verify that columns_with_statistics is non-zero for all critical filter columns. Without column statistics, query engines cannot perform predicate pushdown and will scan every row group regardless of your spatial filter.

#Production-Ready Implementation

The following self-contained module combines profiling, sizing, writing, and validation into a single pipeline function suitable for ETL jobs and CI integration:

# pyarrow>=12.0, geopandas>=0.13, zstandard>=0.21
from __future__ import annotations

import math
import time
import logging
from dataclasses import dataclass
from pathlib import Path

import geopandas as gpd
import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq

logger = logging.getLogger(__name__)


@dataclass
class RowGroupConfig:
    """Resolved row group parameters for a specific dataset and engine target."""
    target_mb: int
    rows_per_group: int
    bytes_per_row_mean: float
    geom_bytes_p95: float


@dataclass
class WriteResult:
    """Summary of a completed Parquet write for logging and alerting."""
    path: str
    num_row_groups: int
    total_rows: int
    compressed_mb_min: float
    compressed_mb_max: float
    compressed_mb_mean: float
    columns_with_statistics: int
    write_seconds: float


def resolve_row_group_config(
    gdf: gpd.GeoDataFrame,
    target_mb: int = 128,
    sample_n: int = 5_000,
    min_rows: int = 5_000,
    max_rows: int = 2_000_000,
) -> RowGroupConfig:
    """
    Profile gdf and compute a safe row count for the given uncompressed MB target.

    Raises:
        ValueError: If gdf is empty or geometry column returns zero byte footprint.
    """
    if gdf.empty:
        raise ValueError("GeoDataFrame is empty — cannot profile row footprint.")

    sample = gdf if len(gdf) <= sample_n else gdf.sample(n=sample_n, random_state=42)
    geom_col = sample.geometry.name

    attr_bytes = (
        sample.drop(columns=[geom_col])
        .memory_usage(deep=True, index=False)
        .sum()
    )
    attr_per_row = attr_bytes / len(sample)

    wkb_sizes = sample.geometry.apply(lambda g: len(g.wkb) if g is not None else 0)
    geom_mean = float(wkb_sizes.mean())
    geom_p95 = float(np.percentile(wkb_sizes, 95))

    bytes_per_row = attr_per_row + geom_mean
    if bytes_per_row <= 0:
        raise ValueError(f"bytes_per_row is non-positive ({bytes_per_row}); check geometry column.")

    if geom_p95 > 3 * geom_mean:
        logger.warning(
            "Geometry p95 (%.0f B) is >3× the mean (%.0f B). "
            "Consider splitting by geometry type or simplifying outliers before writing.",
            geom_p95,
            geom_mean,
        )

    target_bytes = target_mb * 1024 * 1024
    raw_count = int(math.ceil(target_bytes / bytes_per_row))
    rows_per_group = max(min_rows, min(raw_count, max_rows))

    return RowGroupConfig(
        target_mb=target_mb,
        rows_per_group=rows_per_group,
        bytes_per_row_mean=bytes_per_row,
        geom_bytes_p95=geom_p95,
    )


def write_sized_geoparquet(
    gdf: gpd.GeoDataFrame,
    output_path: str | Path,
    target_mb: int = 128,
    compression_level: int = 3,
    sample_n: int = 5_000,
) -> WriteResult:
    """
    Write a GeoDataFrame to Parquet with profiled row group sizing and ZSTD compression.

    Args:
        gdf: Input GeoDataFrame with representative geometry distribution.
        output_path: Destination path (local or s3://bucket/key).
        target_mb: Desired uncompressed size per row group in megabytes.
        compression_level: ZSTD level 1–22. Level 3 balances ratio and CPU cost.
        sample_n: Rows to sample during profiling. Use >=5 000 for mixed geometry.

    Returns:
        WriteResult dataclass with post-write validation metrics.

    Raises:
        ValueError: If gdf is empty or geometry footprint cannot be determined.
    """
    config = resolve_row_group_config(gdf, target_mb=target_mb, sample_n=sample_n)
    logger.info(
        "Writing with row_group_size=%d rows (~%d MB target, %.0f B/row mean)",
        config.rows_per_group,
        config.target_mb,
        config.bytes_per_row_mean,
    )

    table = pa.Table.from_pandas(gdf, preserve_index=False)
    t0 = time.perf_counter()

    pq.write_table(
        table,
        str(output_path),
        row_group_size=config.rows_per_group,
        use_dictionary=True,
        write_statistics=True,
        compression="zstd",
        compression_level=compression_level,
    )

    elapsed = time.perf_counter() - t0

    meta = pq.read_metadata(str(output_path))
    rg_sizes = [meta.row_group(i).total_byte_size for i in range(meta.num_row_groups)]
    stats_present = sum(
        1
        for i in range(meta.num_row_groups)
        for j in range(meta.row_group(i).num_columns)
        if meta.row_group(i).column(j).is_stats_set
    )

    result = WriteResult(
        path=str(output_path),
        num_row_groups=meta.num_row_groups,
        total_rows=meta.num_rows,
        compressed_mb_min=round(min(rg_sizes) / 1024 / 1024, 2),
        compressed_mb_max=round(max(rg_sizes) / 1024 / 1024, 2),
        compressed_mb_mean=round(sum(rg_sizes) / len(rg_sizes) / 1024 / 1024, 2),
        columns_with_statistics=stats_present,
        write_seconds=round(elapsed, 2),
    )
    logger.info("Write complete: %s", result)
    return result

#Benchmark Reference Matrix

Results measured on a 5 million-feature land parcel dataset (mixed polygon complexity, EPSG:4326, geopandas 0.13, pyarrow 14, ZSTD level 3, MacBook M2 Pro 16 GB):

Key observations:

• Compression ratio plateaus above 128 MB — larger groups add write complexity without storage benefit.
• DuckDB optimal is 64–128 MB; beyond that, memory pressure from loading an entire group degrades parallelism.
• Athena optimal is 128–256 MB; smaller groups multiply GET request counts and inflate scan cost.
• Write time differences are marginal across all candidates — do not optimise write time at the expense of query time.

#Failure Modes and Gotchas

Dictionary bloat at large row group sizes. When a row group exceeds ~512 MB uncompressed and contains a high-cardinality string column (feature IDs, address strings), PyArrow’s dictionary builder may exceed its 2 GB limit and silently fall back to plain encoding for that column. Monitor use_dictionary effectiveness by checking compressed column sizes in the metadata; a plain-encoded column will be substantially larger than its dictionary-encoded equivalent.

WKB size variance overwhelming the row count estimate. If your GeoDataFrame contains both simple point geometries and multi-polygon administrative boundaries in the same file, the mean WKB size is a poor predictor. The 95th-percentile estimate from resolve_row_group_config is more conservative and prevents row groups from ballooning to 5–10× your target. Consider splitting heterogeneous geometry datasets by type before writing.

Missing statistics blocking predicate pushdown. If write_statistics=False (the PyArrow default in some older versions), query engines cannot read column min/max values and must decompress every row group to evaluate spatial filters. Always explicitly set write_statistics=True and confirm it worked by checking columns_with_statistics in the WriteResult.

Splitting multi-part polygons across row group boundaries. Parquet does not track topology relationships across row groups. If you have multi-part geometries represented as separate rows with a shared parent ID, sorting by that ID before writing ensures all parts land in the same or adjacent groups — preventing topology validation failures in downstream GIS tools. Run a spatial sort by Z-order or Hilbert curve first when your workload includes polygon assembly operations.

Ignoring file-level row group count in partitioned datasets. When writing multiple files into a Hive-partitioned layout, each file’s row group configuration is independent. A partition shard with few features may produce a single undersized row group (<10 MB), inflating metadata and S3 HEAD request overhead. Add a minimum row threshold guard (min_rows=5_000) and merge small shards before writing.

#FAQ

Q: What is the difference between row group size and page size in Parquet?

Row groups contain column chunks, which are further subdivided into data pages (default 1 MB per page). Page size controls read granularity within a column chunk and affects null-bitmap overhead. For geospatial workloads, row group size is the primary performance lever; page size rarely needs tuning unless you have extremely sparse columns with many nulls.

Q: Should I sort by spatial index before setting row group size?

Yes. Sorting by a spatial index (Z-order, Hilbert curve, or quadtree partition ID) before writing clusters spatially adjacent features into the same row groups. This maximises the effectiveness of bounding-box predicate pushdown — a spatial filter can skip entire groups rather than scanning every group for matching coordinates. Set row group size first, then apply the sort; the sort does not change the optimal byte budget.

Q: Why does PyArrow’s row_group_size parameter take a row count instead of bytes?

Parquet’s physical layout is determined at write time, before compression, so byte budgets require knowing the uncompressed footprint in advance. PyArrow exposes row counts as the API contract and delegates byte estimation to the caller. The workflow above (profile → convert → write) is the standard pattern for production pipelines where geometry size varies.

Q: How do row groups interact with Parquet file partitioning?

They are independent dimensions. File partitioning (by date, region, or spatial tile) controls which rows land in which files. Row groups control the internal chunking within each file. Both need tuning for geospatial workloads: partitioning limits scan scope at the file level; row groups limit scan scope at the column-chunk level within a file.

#Related

• Tuning Row Group Size for Cloud Query Performance
• ZSTD Compression Levels for Geospatial Data
• Dictionary Encoding for Categorical GIS Attributes
• Spatial Partitioning with Quadtree Indexes
• Understanding Parquet Columnar Storage for GIS

← Back to Compression, Chunking & Spatial Indexing