#Dictionary Encoding for Categorical GIS Attributes

Geospatial datasets routinely carry categorical attributes with extreme value repetition: land cover classifications, administrative zone codes, sensor types, soil taxonomy, and zoning designations. Stored as raw UTF-8 strings or unoptimized integers, these columns consume disproportionate memory, inflate I/O during cloud retrieval, and force query engines to perform expensive string comparisons on every predicate evaluation. Dictionary encoding solves this by replacing repeated values with compact integer indices while maintaining a single shared lookup table—the dictionary—per row group. When paired with the columnar layout of GeoParquet and the in-memory Apache Arrow type system, this technique routinely cuts storage footprints by 60–90% and accelerates query execution through vectorized index comparisons instead of sequential string scans. It sits within the broader Compression, Chunking & Spatial Indexing strategy as the semantic layer that makes byte-level codecs far more effective, and it pairs naturally with row group sizing and ZSTD compression downstream.

Crucially, dictionary encoding is a semantic transformation, not a byte-level codec: it preserves exact categorical meaning while priming the data for downstream algorithms that exploit highly repetitive integer sequences far more efficiently than variable-length strings. The rest of this guide covers when the technique pays off, how to apply it correctly through an explicit Arrow schema, and the failure modes that silently disable it.

#Prerequisites

Before implementing dictionary encoding in production pipelines, verify that your stack explicitly supports Apache Arrow dictionary types and columnar I/O semantics:

• Python 3.10+ with pyarrow>=14.0.0 and geopandas>=1.0.0
• Parquet/GeoParquet writer configured with explicit dictionary encoding controls (not relying on implicit type inference)
• Cloud storage SDK (AWS S3, GCS, or Azure Blob) for testing chunked multi-part writes under realistic network latency
• Memory profiling tools: tracemalloc for peak RSS tracking, memory_profiler for line-level allocation in ingestion scripts
• Familiarity with the Apache Arrow Dictionary Encoding model—specifically how dictionary pages are scoped per row group and how index type width (int8, int16, int32) affects memory layout

Dictionary encoding is format-agnostic but performs optimally when the storage engine supports dictionary-aware page layouts. Parquet natively implements this through PLAIN_DICTIONARY (legacy, deprecated) and RLE_DICTIONARY (modern default). Both are automatically selected when Arrow dictionaries are written correctly, but explicit schema declaration prevents silent fallbacks to plain string encoding.

#Architectural Foundations

Dictionary encoding decouples storage representation from value semantics. Instead of writing "Forest", "Forest", "Urban", "Forest", "Wetland" five times to disk, the Parquet writer emits a compact dictionary page containing ["Forest", "Urban", "Wetland"] followed by an index array [0, 0, 1, 0, 2]. The RLE_DICTIONARY encoding then compresses consecutive identical indices into (value, count) pairs, so a region entirely covered in "Forest" records as a single run rather than millions of repeated zero bytes.

This dual-layer approach maps directly to the columnar access pattern dominant in analytical GIS workloads. Filtering land_cover == 'Forest' becomes an integer equality check (index == 0) rather than a full string scan. Group-by operations, spatial joins on categorical boundaries, and aggregations all benefit from reduced memory bandwidth and improved CPU cache locality because the dictionary is materialised once per row group and shared across all index arrays in that group.

When combined with spatial data layouts—spatial partitioning with quadtree indexes or Hilbert-curve ordering—dictionary encoding reinforces those gains: spatially adjacent features in the same quadrant tend to share identical categorical attributes, which raises the RLE run lengths and further compresses index arrays.

#Step-by-Step Workflow

#Step 1 — Profile Candidate Columns and Measure Repetition Ratios

Survey your dataset to locate categorical attributes with cardinality typically below 10,000 unique values. Columns with a repetition ratio above 80% yield the highest compression gains. In municipal GIS workflows, zoning_type, soil_class, land_use_category, and admin_level are prime candidates. Columns with near-unique values (UUIDs, precise sensor IDs, high-resolution timestamps) must be excluded—the dictionary page will consume more space than the raw data.

# pyarrow>=14.0.0, geopandas>=1.0.0
import pandas as pd


def evaluate_column_cardinality(
    df: pd.DataFrame,
    min_ratio: float = 0.80,
    max_cardinality: int = 10_000,
) -> dict[str, dict[str, float | int]]:
    """Return dict of columns that are strong dictionary encoding candidates."""
    results: dict[str, dict[str, float | int]] = {}
    for col in df.select_dtypes(include=["object", "category"]).columns:
        n_rows = len(df)
        if n_rows == 0:
            continue
        n_unique = df[col].nunique(dropna=False)
        repetition_ratio = 1.0 - (n_unique / n_rows)
        if repetition_ratio >= min_ratio and n_unique <= max_cardinality:
            results[col] = {
                "cardinality": n_unique,
                "repetition_ratio": round(repetition_ratio, 4),
                "estimated_saving_pct": round(repetition_ratio * 100, 1),
            }
    return results

#Step 2 — Cast Columns to Arrow DictionaryType with Explicit Schema

Explicitly cast pandas/GeoPandas categorical columns to pyarrow.DictionaryType. Avoid implicit casting during to_parquet(), which often defaults to STRING encoding and nullifies dictionary benefits. Pass a fully declared Arrow schema to pa.Table.from_pandas() to guarantee type preservation through the writer.

# pyarrow>=14.0.0, geopandas>=1.0.0
import pyarrow as pa
import geopandas as gpd
import pandas as pd


def build_dictionary_schema(
    gdf: gpd.GeoDataFrame,
    categorical_cols: list[str],
    index_type: pa.DataType = pa.int32(),
) -> pa.Schema:
    """Build a PyArrow schema with DictionaryType for specified columns."""
    fields: list[pa.Field] = []
    for col in gdf.columns:
        if col == "geometry":
            # GeoParquet writes geometry as WKB large_binary
            fields.append(pa.field(col, pa.large_binary()))
        elif col in categorical_cols:
            dict_type = pa.dictionary(
                index_type=index_type,
                value_type=pa.string(),
                ordered=False,
            )
            fields.append(pa.field(col, dict_type))
        else:
            try:
                fields.append(pa.field(col, pa.from_numpy_dtype(gdf[col].dtype)))
            except NotImplementedError:
                fields.append(pa.field(col, pa.string()))
    return pa.schema(fields)


def cast_to_dictionary(
    gdf: gpd.GeoDataFrame,
    categorical_cols: list[str],
) -> pa.Table:
    """Convert a GeoDataFrame to a PyArrow Table with dictionary-encoded columns."""
    schema = build_dictionary_schema(gdf, categorical_cols)
    return pa.Table.from_pandas(gdf, schema=schema, preserve_index=False)

#Step 3 — Align Row Group Boundaries to Dictionary Scope

Dictionaries in Parquet are scoped per row group. When row group boundaries are too small, identical dictionaries are duplicated across groups, adding metadata bloat and inflating file sizes. When they are too large, Arrow must materialise the entire dictionary and its index array in memory simultaneously—a problem for datasets with many millions of rows. Consult the row group sizing strategies for Parquet guidance before choosing row_group_size.

For most municipal or regional GIS datasets with fewer than 50 unique categorical values, a row group size of 250,000–500,000 rows balances dictionary reuse against memory pressure during cloud reads. Very high-cardinality columns (thousands of unique codes) may warrant smaller row groups to keep dictionary pages under 64 MB.

# pyarrow>=14.0.0
import pyarrow as pa
import pyarrow.parquet as pq
from pathlib import Path


def write_dictionary_encoded_parquet(
    table: pa.Table,
    output_path: str | Path,
    row_group_size: int = 500_000,
    compression: str = "zstd",
    compression_level: int = 3,
) -> None:
    """Write a dictionary-encoded PyArrow Table to Parquet with ZSTD compression."""
    pq.write_table(
        table,
        str(output_path),
        row_group_size=row_group_size,
        use_dictionary=True,          # honour DictionaryType fields
        compression=compression,
        compression_level=compression_level,
        write_statistics=True,        # enables predicate pushdown on index columns
        data_page_size=1_048_576,     # 1 MB data pages for cloud prefetch alignment
    )

#Step 4 — Apply Secondary Byte-Level Compression

Dictionary encoding transforms repetitive strings into compact integers, but it does not replace byte-level compression. Layer ZSTD compression on top of dictionary-encoded pages to compress the integer index runs and the dictionary string values themselves. ZSTD level 3 is the recommended default for categorical GIS attributes: it achieves near-maximum compression on highly repetitive integer sequences while remaining fast enough for real-time decompression in cloud query engines like DuckDB and AWS Athena.

Snappy is an alternative when decompression throughput is critical and storage cost is secondary. LZ4 suits streaming ingestion pipelines where write throughput dominates. For a systematic comparison across all three codecs on vector workloads, see the optimal ZSTD levels for vector vs raster data detail guide.

#Step 5 — Validate Encoding Metadata

After writing, confirm that RLE_DICTIONARY appears in the encoding list for each expected column. A column that fell back to PLAIN encoding was not dictionary-encoded—this typically means the Arrow schema was not honoured by the writer, or cardinality exceeded the writer’s internal dictionary memory threshold during the row group write.

# pyarrow>=14.0.0
import pyarrow.parquet as pq


def validate_dictionary_encoding(path: str) -> None:
    """Print encoding metadata for every column in every row group."""
    pf = pq.ParquetFile(path)
    for rg_idx in range(pf.metadata.num_row_groups):
        rg = pf.metadata.row_group(rg_idx)
        print(f"\n--- Row Group {rg_idx} ({rg.num_rows:,} rows) ---")
        for col_idx in range(rg.num_columns):
            col = rg.column(col_idx)
            status = (
                "OK (dict)"
                if "RLE_DICTIONARY" in col.encodings
                else "WARNING: no dictionary encoding"
            )
            print(
                f"  {col.path_in_schema:<30} "
                f"encodings={col.encodings}  "
                f"compressed={col.total_compressed_size:,} B  "
                f"[{status}]"
            )

#Production-Ready Implementation

The following module combines all four steps into a production pipeline with structured error handling, logging, and per-column encoding validation. It is designed as a drop-in processing step for GeoParquet export pipelines.

# pyarrow>=14.0.0, geopandas>=1.0.0, loguru>=0.7.0
from __future__ import annotations

import tracemalloc
from pathlib import Path

import geopandas as gpd
import pyarrow as pa
import pyarrow.parquet as pq
from loguru import logger


def _detect_categorical_columns(
    gdf: gpd.GeoDataFrame,
    min_ratio: float = 0.80,
    max_cardinality: int = 10_000,
) -> list[str]:
    candidates: list[str] = []
    n_rows = len(gdf)
    if n_rows == 0:
        return candidates
    for col in gdf.select_dtypes(include=["object", "category"]).columns:
        n_unique = gdf[col].nunique(dropna=False)
        ratio = 1.0 - (n_unique / n_rows)
        if ratio >= min_ratio and n_unique <= max_cardinality:
            logger.info(
                f"  {col}: {n_unique} unique values, {ratio:.1%} repetition "
                f"→ dictionary encoding candidate"
            )
            candidates.append(col)
    return candidates


def encode_and_write(
    gdf: gpd.GeoDataFrame,
    output_path: str | Path,
    row_group_size: int = 500_000,
    compression_level: int = 3,
    categorical_cols: list[str] | None = None,
) -> dict[str, object]:
    """
    Encode categorical GIS attributes as Arrow dictionaries and write to GeoParquet.

    Returns a summary dict with file size, row groups, and detected columns.
    """
    output_path = Path(output_path)
    if not output_path.parent.exists():
        raise FileNotFoundError(f"Output directory does not exist: {output_path.parent}")

    if categorical_cols is None:
        logger.info("Auto-detecting dictionary encoding candidates…")
        categorical_cols = _detect_categorical_columns(gdf)

    if not categorical_cols:
        logger.warning("No categorical columns detected; writing with default encoding.")

    # Build schema and cast
    fields: list[pa.Field] = []
    for col in gdf.columns:
        if col == "geometry":
            fields.append(pa.field(col, pa.large_binary()))
        elif col in categorical_cols:
            fields.append(
                pa.field(col, pa.dictionary(pa.int32(), pa.string(), ordered=False))
            )
        else:
            try:
                fields.append(pa.field(col, pa.from_numpy_dtype(gdf[col].dtype)))
            except NotImplementedError:
                fields.append(pa.field(col, pa.string()))

    schema = pa.schema(fields)

    tracemalloc.start()
    try:
        table = pa.Table.from_pandas(gdf, schema=schema, preserve_index=False)
        pq.write_table(
            table,
            str(output_path),
            row_group_size=row_group_size,
            use_dictionary=True,
            compression="zstd",
            compression_level=compression_level,
            write_statistics=True,
            data_page_size=1_048_576,
        )
        _, peak_bytes = tracemalloc.get_traced_memory()
    finally:
        tracemalloc.stop()

    # Validate
    pf = pq.ParquetFile(str(output_path))
    rg0 = pf.metadata.row_group(0)
    encoding_report: dict[str, list[str]] = {}
    for i in range(rg0.num_columns):
        col_meta = rg0.column(i)
        encoding_report[col_meta.path_in_schema] = list(col_meta.encodings)

    file_bytes = output_path.stat().st_size
    logger.success(
        f"Wrote {output_path.name}: {file_bytes / 1_048_576:.1f} MB, "
        f"{pf.metadata.num_row_groups} row groups, "
        f"peak RSS {peak_bytes / 1_048_576:.1f} MB"
    )
    return {
        "output_path": str(output_path),
        "file_size_mb": round(file_bytes / 1_048_576, 2),
        "num_row_groups": pf.metadata.num_row_groups,
        "peak_rss_mb": round(peak_bytes / 1_048_576, 2),
        "dictionary_columns": categorical_cols,
        "encoding_report": encoding_report,
    }

#Benchmark Reference Matrix

The table below reflects measurements on a 10-million-row municipal parcel dataset with 12 categorical columns (7 qualifying as dictionary candidates, max cardinality 148). All figures are averages across three runs; query latency is DuckDB 0.10 local, not cloud.

Dictionary encoding contributes the majority of storage reduction. ZSTD compresses the resulting integer arrays efficiently without meaningfully increasing query latency at level 3.

#Failure Modes and Gotchas

Silent fallback to plain encoding. When pd.DataFrame.to_parquet() is called without an explicit schema, pandas infers STRING type for object columns and ignores the use_dictionary=True flag at the column level. Always declare the Arrow schema explicitly; never rely on implicit type inference for encoding-critical columns.

Dictionary page exceeds writer memory threshold. Parquet writers enforce a maximum dictionary page size (commonly 8–16 MB). If unique values × average string length exceeds that threshold during a row group write, the engine abandons dictionary encoding mid-group and falls back to plain. Validate encoding metadata after every write; if fallbacks appear, reduce row_group_size or pre-filter the high-cardinality column.

Misaligned row group boundaries break dictionary reuse. Writing small row groups (fewer than ~50,000 rows) while the column has very few unique values causes the dictionary to be serialised repeatedly across hundreds of groups. Each duplicate adds ~100–500 bytes of overhead per column. Match row_group_size to the access pattern: larger groups for analytics, smaller groups for mixed random access. Refer to row group sizing strategies for the sizing formula.

Applying dictionary encoding to near-unique columns. UUID primary keys, precise GPS timestamps, or high-resolution sensor IDs have cardinality approaching row count. Encoding these produces a dictionary page larger than the raw column data, increases index array overhead, and degrades every read path. Exclude these columns explicitly in categorical_cols and let them encode as plain strings or typed integers.

Cross-partition dictionary resolution overhead in distributed engines. In federated query engines (DuckDB, Polars, Spark), a filter like country_code == 'NZ' must resolve the string against each partition’s dictionary before comparing indices. For datasets partitioned across thousands of files, this dictionary-resolution fan-out adds measurable latency. Mitigate by pre-normalising string dictionaries to a shared integer foreign-key reference table, or by using Hive-style partition columns (folder-level predicates) to prune partitions before dictionary resolution occurs.

CRS attribute columns mis-treated as categoricals. EPSG code columns like crs_auth_name look categorical ("EPSG", repeated billions of times) but are structural metadata, not analytical dimensions. Encoding them as dictionaries is fine technically, but treating them as filterable attributes can mislead query planners. Keep CRS metadata in file-level key-value metadata, not as row-level categorical columns.

#FAQ

What cardinality threshold makes dictionary encoding worthwhile?

Columns with fewer than 10,000 unique values and a repetition ratio above 80% yield the strongest gains. Below that cardinality the dictionary page is tiny, index arrays are highly compressible, and you see 60–90% size reductions. Above roughly 65,000 unique values the index type remains efficient (int32), but dictionary page size grows linearly and Parquet engines may fall back to plain encoding if memory thresholds are exceeded during write.

Does dictionary encoding replace ZSTD or Snappy compression?

No—they operate at different layers. Dictionary encoding is a semantic transformation that replaces repeated strings with compact integers. Byte-level codecs like ZSTD then compress those integer index arrays and the dictionary page itself, typically adding another 30–50% size reduction on top of the encoding step. Skipping secondary compression leaves significant space on the table.

Why does dictionary scope break across Parquet row group boundaries?

The Parquet specification scopes each dictionary page to a single row group. When a table is split across multiple row groups, each writes its own dictionary page. For columns with the same finite value set, this means the dictionary is duplicated rather than shared, adding metadata overhead. Tuning row group size to balance dictionary reuse against memory pressure during reads is therefore a key part of the encoding strategy.

Can I apply dictionary encoding when writing GeoParquet from GeoPandas?

Yes, but you must declare an explicit Arrow schema before calling to_parquet(). Implicit conversion through GeoPandas/pandas defaults to STRING encoding for object columns, silently bypassing dictionary pages. Casting categorical columns to pa.dictionary(pa.int32(), pa.string()) in a pyarrow schema and using pa.Table.from_pandas() with that schema guarantees the correct encoding reaches the Parquet writer.

Related

• Row Group Sizing Strategies for Parquet — size row groups to maximise dictionary reuse and cloud I/O efficiency
• Tuning Row Group Size for Cloud Query Performance — benchmark-driven sizing formulas for S3 and GCS workloads
• ZSTD Compression Levels for Geospatial Data — secondary compression layered on top of dictionary encoding
• Spatial Partitioning with Quadtree Indexes — align partition boundaries with categorical attribute distributions
• Optimal ZSTD Levels for Vector vs Raster Data — codec selection for mixed vector/raster pipelines

← Back to Compression, Chunking & Spatial Indexing