#WKB vs Native Arrow Geometry Encoding

WKB is an opaque blob to everything below the geometry library: the storage format sees bytes with a length, so it cannot apply numeric encodings, cannot compute meaningful statistics, and cannot skip anything. A native Arrow geometry encoding stores coordinates as typed child arrays, which makes them visible to exactly that machinery. The size difference is worth ten to twenty-five per cent; the decode-cost difference is larger. The portability difference is why WKB is still the right thing to publish. This page continues geometry encoding and coordinate precision.

#Quick Reference

#What “Opaque” Costs

A WKB polygon is: one byte of endianness, four bytes of geometry type, four bytes of ring count, then for each ring four bytes of point count followed by the ordinates. Parquet stores that entire structure as one value in a BYTE_ARRAY column. Everything Parquet knows how to do with numbers — delta encoding, byte-stream splitting, dictionary encoding, min/max statistics — is unavailable, because it is not looking at numbers.

A native Arrow encoding instead represents a polygon column as a nested structure: a list of rings, each a list of points, each point a pair of double child arrays. The x ordinates of every vertex in the file live in one contiguous numeric array; likewise the y ordinates. Now the format’s encodings apply directly, and they apply well, because x values within a region are all similar and so are y values — the very property that byte-stream splitting exists to exploit.

There is a second effect that is easy to overlook. Separating x from y improves compression on its own, because an interleaved stream alternates between two different value distributions, and a compressor modelling byte position sees noise where a separated stream sees two smooth runs. This is the same mechanism as the shuffle filter described in Blosc codec selection for Zarr, applied to vector geometry.

#Writing Both

# Requires: geopandas>=1.0, pyarrow>=16, geoarrow-pyarrow>=0.1  (Python 3.10+)
from __future__ import annotations

from pathlib import Path

import geoarrow.pyarrow as ga
import geopandas as gpd
import pyarrow as pa
import pyarrow.parquet as pq


def write_wkb(frame: gpd.GeoDataFrame, target: Path, **kwargs) -> int:
    """The interchange copy: readable by every tool that speaks GeoParquet."""
    frame.to_parquet(target, compression="zstd", compression_level=3, **kwargs)
    return target.stat().st_size


def write_native(frame: gpd.GeoDataFrame, target: Path) -> int:
    """The internal copy: coordinates as typed child arrays."""
    if frame.crs is None:
        raise ValueError("a CRS is required before writing geometry")

    attributes = pa.Table.from_pandas(
        frame.drop(columns=frame.geometry.name), preserve_index=False
    )
    geometry = ga.as_geoarrow(frame.geometry.to_wkb())
    table = attributes.append_column("geometry", geometry)

    pq.write_table(
        table, target,
        compression="zstd", compression_level=3,
        # BYTE_STREAM_SPLIT reorders the bytes of each double so that like
        # significance sits together — only possible because x and y are now
        # real numeric columns rather than bytes inside a blob.
        column_encoding={"geometry.list.list.x": "BYTE_STREAM_SPLIT",
                         "geometry.list.list.y": "BYTE_STREAM_SPLIT"},
        use_dictionary=False,
        write_statistics=True,
    )
    return target.stat().st_size


def compare(frame: gpd.GeoDataFrame, out_dir: Path) -> dict[str, object]:
    """Write both and report the difference that actually matters."""
    out_dir.mkdir(parents=True, exist_ok=True)
    wkb_bytes = write_wkb(frame, out_dir / "wkb.parquet")
    native_bytes = write_native(frame, out_dir / "native.parquet")
    return {
        "wkb_mb": round(wkb_bytes / 1e6, 1),
        "native_mb": round(native_bytes / 1e6, 1),
        "saving_pct": round(100 * (1 - native_bytes / wkb_bytes), 1),
    }

#Validation

Compare compressed sizes and decode time, not raw sizes. The decode difference is often the more valuable of the two and is invisible on disk.

# Requires: pyarrow>=16, shapely>=2.0 — decode cost, measured
import time
import pyarrow.parquet as pq
from shapely import from_wkb

start = time.perf_counter()
wkb_col = pq.read_table("wkb.parquet", columns=["geometry"])["geometry"]
geoms = from_wkb(wkb_col.to_pylist())          # every blob parsed individually
wkb_seconds = time.perf_counter() - start

start = time.perf_counter()
native = pq.read_table("native.parquet", columns=["geometry"])
xs = native["geometry"].combine_chunks().flatten().flatten().field("x")
native_seconds = time.perf_counter() - start   # coordinates are already an array

print(f"WKB decode {wkb_seconds:.2f}s · native {native_seconds:.2f}s")

Expected ranges on a 4 million-feature polygon layer: native encoding 12–20% smaller compressed, and coordinate access three to eight times faster because there is no per-feature parse. Attribute-only queries show no difference, because neither encoding is touched.

#Edge Cases and Caveats

A reader that silently falls back. Some tools accept a native-encoded GeoParquet, fail to recognise the geometry column, and expose it as a nested structure of numbers rather than as geometry. Nothing errors; the layer simply has no geometry. Test with each consumer you actually have before switching a published dataset.

Mixed geometry types in one column. Native encodings are typed — a polygon array is a polygon array — so a column holding points, lines, and polygons cannot use the most efficient representation and falls back to something closer to WKB. Splitting by type is usually the right answer anyway, as mapping mixed geometry types argues.

Statistics that still are not spatial. Native encoding gives x and y real min/max statistics, but those describe the column, not per-feature bounding boxes, and engines generally do not use them for spatial pruning. Covering bbox columns remain necessary either way.

#Frequently Asked Questions

What is the practical difference between WKB and a native Arrow geometry encoding?

WKB stores each geometry as a variable-length binary blob in a single column, so from the storage layer’s point of view it is opaque bytes with a length. A native Arrow encoding stores the coordinates as actual numeric child arrays inside nested list structures, so x and y are typed columns the format can apply its own encodings and statistics to. The consequence is that WKB is universally readable and encoding-opaque, while native Arrow is more selective about readers and far more transparent to the machinery underneath.

Does native Arrow encoding produce smaller files?

Usually, by ten to twenty-five per cent after compression on typical vector layers, and more on dense linework. The reason is that separating x from y groups similar magnitudes together, and Parquet can then apply delta and byte-stream-split encodings that it cannot apply to an opaque blob. It also removes the per-geometry WKB header — a type code and endianness flag repeated on every feature.

Should I stop writing WKB?

Not for anything you publish. WKB is the interchange encoding every tool reads, and a file nobody can open is worth nothing regardless of its size. The pragmatic split is to publish WKB for external consumers and keep an internally-read copy in a native Arrow encoding where decode cost and size matter, accepting the duplication as the price of serving both audiences well.

#Related

• Geometry Encoding and Coordinate Precision — parent guide: precision, delta encoding, and type narrowing
• Coordinate Precision Reduction for Smaller GeoParquet — the lever that compounds with this one
• Understanding Parquet Columnar Storage for GIS — how the format encodes a typed column
• Dictionary Encoding for Categorical GIS Attributes — the same visibility argument applied to attributes

← Back to Geometry Encoding and Coordinate Precision