#ZSTD vs Snappy vs LZ4 for GeoParquet

For GeoParquet on cloud object storage, make ZSTD your default codec; keep Snappy for transient shuffle and spill files, and reserve LZ4 for hot data read repeatedly from local NVMe. The reasoning is a single trade-off: ZSTD gives the best compression ratio, so it moves the fewest bytes across the network, and its decompression is still fast enough that it never becomes the bottleneck on a remote read. Snappy and LZ4 compress less but decode faster, which only helps when the data path is local and network transfer is not the limiting factor. This page compares the three codecs head to head on ratio, throughput, engine support, and memory, and gives you a harness to confirm the defaults on your own data. It narrows the level-selection guidance in ZSTD Compression Levels for Geospatial Data down to the codec choice itself.

#Quick-Reference Comparison

#Why ZSTD Is the Cloud Default

On a remote read the cost that dominates is bytes over the wire. Object-storage bandwidth per connection typically lands well below a codec’s decode speed, so the codec that ships fewer bytes wins even if it decodes a little slower. ZSTD at level 3 compresses vector GeoParquet 30–45% tighter than Snappy, which means proportionally fewer bytes fetched from S3 per bounding-box scan, and it decodes at roughly 800 MB/s per core — faster than the network delivers those bytes. The net effect is lower query latency and lower egress cost at the same time. This is why the GeoParquet ecosystem and engines such as DuckDB treat ZSTD as the sensible baseline. The tighter ratio also compounds with a curve-clustered layout: once rows are sorted by a space-filling curve, neighbouring geometries share coordinate prefixes and ZSTD’s deeper search exploits them, so ordering and codec reinforce each other. For the level-by-level ratio and latency curve, and where level 3 sits against archival levels, see optimal ZSTD levels for vector vs raster data.

#Where Snappy and LZ4 Still Win

The picture inverts the moment the network drops out of the path. Snappy is the pragmatic choice for the transient files a query engine spills during a shuffle or a large join: they are written once, read once, live on local disk, and are deleted immediately, so a smaller ratio saves nothing and Snappy’s slightly faster round trip plus universal support make it the low-risk option. LZ4 targets the opposite extreme — data that is genuinely hot, read over and over from local NVMe, where decompression throughput above 2 GB/s per core is the constraint and ZSTD’s extra ratio buys nothing because there is no network to save. Outside those two niches, the ratio advantage of ZSTD wins. One caveat on portability: Parquet defines two LZ4 variants (LZ4 and LZ4_RAW), and older readers disagree on which they accept, so verify LZ4 files open in every engine that must read them before standardising on it. Whichever codec you choose, coordinate it with your row group sizing strategy, since group size governs how much data each decode call must expand.

#Benchmark the Three Codecs

The harness writes the same PyArrow table with each codec, then measures ratio and round-trip time so you can validate the defaults against your own geometry and attribute mix.

# Requires: pyarrow>=14.0
from __future__ import annotations

import time
from pathlib import Path
from typing import Any

import pyarrow as pa
import pyarrow.parquet as pq

# codec name -> (compression, compression_level or None)
CODECS: dict[str, tuple[str, int | None]] = {
    "zstd_l3": ("zstd", 3),
    "snappy": ("snappy", None),
    "lz4": ("lz4", None),
}


def compare_codecs(
    table: pa.Table,
    scratch: Path = Path("/tmp/geoparquet_codec_bench"),
    row_group_size: int = 100_000,
) -> list[dict[str, Any]]:
    """Write `table` with ZSTD, Snappy, and LZ4; report ratio and timings."""
    scratch.mkdir(parents=True, exist_ok=True)
    raw_mb = table.nbytes / 1024**2
    results: list[dict[str, Any]] = []

    for name, (codec, level) in CODECS.items():
        path = scratch / f"bench_{name}.parquet"
        try:
            t0 = time.perf_counter()
            pq.write_table(
                table, str(path),
                compression=codec,
                compression_level=level,   # ignored by snappy/lz4
                row_group_size=row_group_size,
                write_statistics=True,
            )
            write_s = time.perf_counter() - t0

            t0 = time.perf_counter()
            _ = pq.read_table(str(path))
            read_s = time.perf_counter() - t0

            file_mb = path.stat().st_size / 1024**2
            results.append({
                "codec": name,
                "file_mb": round(file_mb, 2),
                "ratio": round(raw_mb / file_mb, 2) if file_mb else None,
                "write_s": round(write_s, 3),
                "read_s": round(read_s, 3),
            })
        except pa.lib.ArrowNotImplementedError as exc:
            results.append({"codec": name, "error": f"unsupported: {exc}"})
        finally:
            if path.exists():
                path.unlink()

    return results


if __name__ == "__main__":
    import geopandas as gpd  # geopandas>=0.14
    tbl = gpd.read_file("parcels.gpkg").to_arrow()
    for row in compare_codecs(tbl):
        print(row)

#Validation

Run the harness and read the ratio and read_s columns together. Expected shape on a mixed vector table (geometry plus categorical attributes):

{'codec': 'zstd_l3', 'file_mb': 42.1, 'ratio': 4.6, 'write_s': 0.51, 'read_s': 0.22}
{'codec': 'snappy',  'file_mb': 78.3, 'ratio': 2.5, 'write_s': 0.44, 'read_s': 0.19}
{'codec': 'lz4',     'file_mb': 82.0, 'ratio': 2.4, 'write_s': 0.39, 'read_s': 0.16}

ZSTD produces the smallest file by a wide margin while its read time stays within a few tens of milliseconds of Snappy and LZ4. On a remote read, multiply the file-size gap by your object-storage transfer time — that delta usually dwarfs the small local decode difference, confirming ZSTD for the cloud path.

#Edge Cases and Caveats

Raster bands inside Parquet. High-entropy raster pixel data barely compresses, so ZSTD’s ratio advantage shrinks and its extra CPU is wasted. For raster-heavy columns, a lower ZSTD level or Snappy is often the better trade — the raster-specific reasoning lives in optimal ZSTD levels for vector vs raster data.

Legacy engine compatibility. A very old reader (Hive 1.x, legacy Impala) may not accept ZSTD or a particular LZ4 variant. Where you must support such consumers, fall back to Snappy, which every Parquet reader understands.

CPU-starved ingest. If your write path is already CPU-bound and write latency is the hard constraint, ZSTD level 3’s encode cost can pinch. Drop to ZSTD level 1, or use Snappy for the ingest tier and re-compress to ZSTD in a later compaction pass.

#Frequently Asked Questions

Which codec should be the default for GeoParquet on cloud storage?

ZSTD at level 3 is the right default for GeoParquet in S3, GCS, or Azure Blob. Its 30 to 45 percent better ratio than Snappy directly cuts the bytes transferred per query, and its decompression throughput of roughly 800 megabytes per second per core comfortably exceeds cloud network bandwidth, so decoding is never the bottleneck on remote reads.

Is Snappy or LZ4 ever the better choice over ZSTD?

Yes. Snappy remains a sensible default for transient shuffle and spill files inside a query engine, where the data is written and read once locally and the smaller ratio does not matter. LZ4 wins for hot data read repeatedly from local NVMe, where its decompression speed above 2 gigabytes per second per core beats ZSTD and the network is not involved.

#Related

• ZSTD Compression Levels for Geospatial Data — parent topic: level selection, benchmarking, and failure modes
• Optimal ZSTD Levels for Vector vs Raster Data — how level interacts with data type once you have chosen ZSTD
• Row Group Sizing Strategies for Parquet — the chunk dimension that governs per-decode work
• Dictionary Encoding for Categorical GIS Attributes — encoding that compounds with any byte-level codec
• Space-Filling Curves for Spatial Partitioning — clustering rows so the codec compresses tighter

← Back to ZSTD Compression Levels for Geospatial Data