#Schema Mapping for Legacy to Modern Geospatial Formats

Migrating geospatial datasets from legacy storage formats to cloud-native architectures has one non-negotiable prerequisite: a precise, deterministic schema translation step. Shapefiles carry implicit type assumptions sampled from the first hundred rows of a DBF file. Legacy PostGIS exports mix geometry subtypes in a single column. Proprietary GDB layers embed coordinate precision in the format itself rather than the field definition. When the target is a compressed columnar format like GeoParquet or FlatGeobuf, any ambiguity in the source schema propagates silently into broken spatial indexes and truncated numeric fields.

Schema mapping is the stage that eliminates that ambiguity. This guide provides a production workflow covering schema discovery, type alignment, spatial normalisation, null routing, serialisation, and evolution strategies — the full set of decisions required to build a reliable migration within a broader Data Conversion & Migration Pipelines architecture.

#Prerequisites

Before implementing schema mapping logic, confirm your environment meets these requirements:

• Python 3.9+ with pip or conda environment isolation
• Core libraries: geopandas>=0.14, pyarrow>=14.0, shapely>=2.0, fiona>=1.9, pyproj>=3.0
• Cloud SDK (optional): boto3 or gcsfs for object-storage staging
• Baseline knowledge: WKT/WKB geometry serialisation, CRS transformation (EPSG/OGC URN), and columnar storage principles
• Test dataset: a representative legacy file (.shp, .gdb layer, or CSV with lat/lon columns) containing mixed types, null geometries, and legacy attribute names

Install dependencies:

pip install geopandas pyarrow shapely fiona pyproj

#Architectural Foundations

The root cause of most migration failures is the mismatch between the loose, row-oriented type model of legacy formats and the strict, columnar model of Apache Arrow and Parquet. In a row-oriented format, each row can carry any value in any field — the format tolerates heterogeneous types per column at read time. Arrow requires every value in a column to conform to a single declared type. When automatic inference bridges that gap, it guesses, and guesses wrong on heterogeneous columns.

The solution is to decouple discovery from mapping. Discovery reads the source without commitment; mapping declares a pa.schema() object that fixes types before a single row is converted. ZSTD compression levels, row group sizing, and spatial index build time are all downstream of this mapping step — if the schema is wrong, those optimisations cannot recover correctness.

The six stages below model the schema mapping pipeline. Each stage produces an immutable artefact — a profile dict, a pa.Schema, a normalised GeoDataFrame, a validated table, a written file, and a versioned manifest — so failures are reproducible and auditable. When a stage raises rather than returns its artefact, hand the failed unit to fallback routing for failed migration jobs so a single malformed source layer never stalls a whole batch.

#Step-by-Step Workflow

#Step 1 — Schema Discovery & Profiling

Legacy formats rarely expose explicit schemas. Shapefiles infer types from the first 100 rows, while legacy databases may use VARCHAR(255) for numeric codes. Begin by extracting raw field names, detected types, geometry types, and CRS metadata. Profile null ratios, string cardinality, and numeric ranges to identify truncation risks before committing to a target schema.

Run discovery as a pre-flight check in any batch conversion pipeline before touching the data. Use geopandas to inspect the DataFrame schema, then cross-reference with pyarrow type inference to flag mismatches early.

import geopandas as gpd
import pandas as pd

def profile_legacy_source(path: str) -> dict:
    """Return a discovery summary: dtypes, geometry info, CRS, null ratios."""
    gdf = gpd.read_file(path)

    null_ratios = (gdf.isnull().sum() / len(gdf)).to_dict()
    geom_types = gdf.geometry.geom_type.value_counts().to_dict()

    return {
        "row_count": len(gdf),
        "dtypes": gdf.dtypes.astype(str).to_dict(),
        "geom_types": geom_types,
        "crs": str(gdf.crs),
        "null_ratios": null_ratios,
    }

profile = profile_legacy_source("legacy_data.shp")
print(profile)

#Step 2 — Type Alignment & Precision Control

Map legacy types to strict Arrow/Parquet equivalents. Avoid implicit casting. Enforce int32/int64 for identifiers, float64 for measurements, and string for categorical attributes. Control decimal precision explicitly to prevent storage bloat and ensure deterministic query results.

Define a pa.schema() object rather than relying on automatic inference. This guarantees consistent column ordering and prevents downstream type coercion errors in distributed query engines like DuckDB or Trino. Dictionary encoding applied to low-cardinality string columns (land use codes, region names) can further reduce output size by 60–80 % on top of the type alignment gains.

import pyarrow as pa

# geopandas>=0.14 / pyarrow>=14.0
target_schema = pa.schema([
    pa.field("id",           pa.int64()),
    pa.field("area_km2",     pa.float64()),
    pa.field("region_name",  pa.string()),
    pa.field("recorded_at",  pa.timestamp("us", tz="UTC")),
    pa.field("geometry",     pa.large_binary()),   # WKB encoding
])

#Step 3 — Spatial Reference & Geometry Normalisation

Legacy datasets frequently mix geometry subtypes (MULTIPOLYGON and POLYGON in the same column) or carry outdated CRS definitions with unofficial authority codes. Normalise to a single geometry type, or explicitly declare mixed-type support in the output format. Reproject to a canonical CRS — EPSG:4326 for global data, a regional projected CRS for analytical workloads requiring planar distance calculations.

The GeoParquet specification mandates explicit CRS metadata in the column schema. Mixed geometry columns must be unified by casting simple types to their Multi* equivalents to maintain query compatibility with spatial index engines. Spatial query performance for downstream consumers depends on consistent geometry typing — see spatial partitioning with quadtree indexes for how geometry homogeneity enables efficient partition pruning.

from shapely.geometry import MultiPolygon

def normalise_geometry_column(gdf, target_crs: str = "EPSG:4326"):
    """Unify Polygon→MultiPolygon and reproject to target_crs in place."""
    gdf = gdf.copy()

    def to_multi(geom):
        if geom is None:
            return None
        if geom.geom_type == "Polygon":
            return MultiPolygon([geom])
        return geom

    gdf["geometry"] = gdf["geometry"].apply(to_multi)
    gdf = gdf.to_crs(target_crs)
    return gdf

#Step 4 — Null Handling & Validation Logic

Missing geometries and incomplete attribute records are endemic in legacy exports. Rather than dropping rows silently or coercing None to POINT EMPTY, implement explicit null routing. Keep null geometries as null WKB. Replace string nulls with pd.NA. Document which columns are permitted to be null in the schema manifest.

For deeper strategies on managing incomplete spatial records, see Handling Null Values in Spatial Schema Mapping. Run post-transformation assertions on row counts, geometry validity, and schema conformity before writing to disk.

def validate_gdf(gdf, expected_crs_epsg: int, input_row_count: int) -> None:
    """Raise AssertionError if any validation check fails."""
    assert len(gdf) == input_row_count, (
        f"Row count mismatch: expected {input_row_count}, got {len(gdf)}"
    )

    invalid_geom = gdf.geometry[gdf.geometry.notna() & ~gdf.geometry.is_valid]
    assert invalid_geom.empty, (
        f"{len(invalid_geom)} invalid geometries detected. "
        "Run gdf.geometry.buffer(0) to attempt auto-repair."
    )

    assert gdf.crs.to_epsg() == expected_crs_epsg, (
        f"CRS mismatch: expected EPSG:{expected_crs_epsg}, got {gdf.crs}"
    )

#Step 5 — Serialisation & Metadata Preservation

Schema mapping extends beyond column types into dataset-level metadata. Field descriptions, provenance tags, coordinate precision notes, and schema version identifiers must be attached during the write phase. GeoParquet and FlatGeobuf support custom metadata dictionaries in the Parquet footer — use them.

For full implementation patterns, see Preserving Metadata During GeoParquet Conversion. Always serialise geometries as WKB rather than WKT to minimise storage footprint and accelerate spatial index construction.

import json
import pyarrow as pa
import pyarrow.parquet as pq

def write_geoparquet(
    gdf,
    output_path: str,
    arrow_schema: pa.Schema,
    geometry_types: list[str],
    compression: str = "snappy",
) -> None:
    """Serialise a GeoDataFrame to GeoParquet with compliant column metadata."""
    gdf = gdf.copy()
    gdf["geometry"] = gdf["geometry"].apply(
        lambda g: g.wkb if g is not None else None
    )

    table = pa.Table.from_pandas(gdf, schema=arrow_schema, preserve_index=False)

    geo_meta = json.dumps({
        "version": "1.0.0",
        "primary_column": "geometry",
        "columns": {
            "geometry": {
                "encoding": "WKB",
                "geometry_types": geometry_types,
            }
        },
    }).encode("utf-8")

    existing = table.schema.metadata or {}
    table = table.replace_schema_metadata({**existing, b"geo": geo_meta})

    pq.write_table(table, output_path, compression=compression)

#Step 6 — Schema Evolution & Pipeline Integration

Schema mapping is rarely a one-time operation. Source systems evolve: legacy vendors deprecate formats, upstream databases rename columns, and downstream consumers request new attributes. Implement schema versioning, drift detection, and backward-compatible evolution to prevent pipeline breakage between runs.

Use pyarrow.unify_schemas() to merge incremental loads. Maintain a manifest file that records schema versions, CRS baselines, and transformation rules. This enables reproducible migrations and simplifies rollbacks when upstream data contracts shift unexpectedly. The diagram below shows the drift detection decision path that runs on each pipeline execution.

import pyarrow as pa
import json
from pathlib import Path

MANIFEST_PATH = Path("schema_manifest.json")

def load_or_init_manifest(schema: pa.Schema, crs_epsg: int) -> dict:
    if MANIFEST_PATH.exists():
        return json.loads(MANIFEST_PATH.read_text())
    manifest = {
        "version": 1,
        "crs_epsg": crs_epsg,
        "fields": {f.name: str(f.type) for f in schema},
    }
    MANIFEST_PATH.write_text(json.dumps(manifest, indent=2))
    return manifest

def detect_schema_drift(current: pa.Schema, manifest: dict) -> list[str]:
    """Return a list of drift messages (empty if schemas match)."""
    issues = []
    for f in current:
        if f.name not in manifest["fields"]:
            issues.append(f"New field: {f.name} ({f.type})")
        elif manifest["fields"][f.name] != str(f.type):
            issues.append(
                f"Type change on '{f.name}': "
                f"was {manifest['fields'][f.name]}, now {f.type}"
            )
    for name in manifest["fields"]:
        if not current.get_field_index(name) >= 0:
            issues.append(f"Removed field: {name}")
    return issues

#Production-Ready Implementation

The function below consolidates all six steps into a single, runnable migration unit with explicit error routing, typed signatures, and structured logging.

import json
import logging
from pathlib import Path

import geopandas as gpd
import pyarrow as pa
import pyarrow.parquet as pq
from shapely.geometry import MultiPolygon

logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
log = logging.getLogger(__name__)


def migrate_legacy_to_geoparquet(
    input_path: str,
    output_path: str,
    target_crs: str = "EPSG:4326",
    id_col: str = "id",
    numeric_cols: list[str] | None = None,
    string_cols: list[str] | None = None,
    compression: str = "snappy",
) -> None:
    """
    Full schema-mapping migration from any GDAL-readable vector format
    to a GeoParquet-compliant Parquet file.

    Requires: geopandas>=0.14, pyarrow>=14.0, shapely>=2.0
    """
    numeric_cols = numeric_cols or []
    string_cols = string_cols or []

    # 1. Discovery
    log.info("Loading legacy dataset: %s", input_path)
    gdf = gpd.read_file(input_path)
    input_row_count = len(gdf)
    log.info("Loaded %d rows; geometry types: %s", input_row_count,
             gdf.geometry.geom_type.value_counts().to_dict())

    # 2. Geometry normalisation
    gdf = gdf.copy()
    gdf["geometry"] = gdf["geometry"].apply(
        lambda g: MultiPolygon([g]) if g and g.geom_type == "Polygon" else g
    )
    gdf = gdf.to_crs(target_crs)

    # 3. Type alignment
    if id_col in gdf.columns:
        gdf[id_col] = gdf[id_col].astype("int64")
    for col in numeric_cols:
        if col in gdf.columns:
            gdf[col] = gdf[col].astype("float64")
    for col in string_cols:
        if col in gdf.columns:
            gdf[col] = gdf[col].astype("string")

    # 4. Validation
    invalid = gdf.geometry[gdf.geometry.notna() & ~gdf.geometry.is_valid]
    if not invalid.empty:
        log.warning("%d invalid geometries — applying buffer(0) repair", len(invalid))
        gdf.loc[invalid.index, "geometry"] = (
            gdf.loc[invalid.index, "geometry"].buffer(0)
        )
    epsg_int = int(target_crs.split(":")[-1])
    assert gdf.crs.to_epsg() == epsg_int, f"CRS validation failed: {gdf.crs}"
    assert len(gdf) == input_row_count, "Row count changed during normalisation"

    # 5. Serialise geometry to WKB
    gdf["geometry"] = gdf["geometry"].apply(
        lambda g: g.wkb if g is not None else None
    )

    # 6. Build explicit Arrow schema
    fields = [pa.field(id_col, pa.int64()), pa.field("geometry", pa.large_binary())]
    fields += [pa.field(c, pa.float64()) for c in numeric_cols if c in gdf.columns]
    fields += [pa.field(c, pa.string()) for c in string_cols if c in gdf.columns]
    arrow_schema = pa.schema(fields)

    table = pa.Table.from_pandas(gdf, schema=arrow_schema, preserve_index=False)

    # 7. Attach GeoParquet column metadata
    geo_meta = json.dumps({
        "version": "1.0.0",
        "primary_column": "geometry",
        "columns": {
            "geometry": {"encoding": "WKB", "geometry_types": ["MultiPolygon"]}
        },
    }).encode("utf-8")
    existing = table.schema.metadata or {}
    table = table.replace_schema_metadata({**existing, b"geo": geo_meta})

    pq.write_table(table, output_path, compression=compression)
    log.info("Migration complete: %s (%d rows)", output_path, len(gdf))


# Example:
# migrate_legacy_to_geoparquet(
#     "legacy.shp", "output.parquet",
#     numeric_cols=["area_km2", "perimeter_m"],
#     string_cols=["region_name", "land_use"],
# )

#Benchmark Reference: Schema Mapping Outcomes

The table below shows measured outcomes from migrating a 4.2 million-row Shapefile containing mixed Polygon/MultiPolygon geometry across a range of configurations. Compression ratios are relative to the source .shp + .dbf footprint on disk.

#Failure Modes and Gotchas

Silent type truncation from Shapefile DBF inference. Shapefiles infer column types from the first 100 rows. A column with all-integer values in the first hundred rows but decimal values in later rows will have its fractional parts silently dropped. Always define pa.schema() explicitly and never let pyarrow infer types from a Shapefile-derived DataFrame.

CRS mismatch before geometry normalisation. Running the Multi* cast after reprojection rather than before can introduce floating-point rounding artefacts in the geometry coordinates. Always normalise geometry types first, then reproject.

Mixed authority codes for the same CRS. Legacy datasets may declare EPSG:4326, OGC:CRS84, or a custom WKT string that maps to the same geodetic coordinate system. pyproj resolves these correctly, but the CRS string in the manifest may differ between runs. Store the normalised crs.to_epsg() integer, not the raw CRS string, in the manifest.

Null geometry rows causing WKB serialisation failures. Calling .wkb on None raises AttributeError. Always guard with lambda g: g.wkb if g is not None else None. Do not drop null-geometry rows without logging their identifiers and count — data loss in a migration must be auditable.

Schema evolution breaking downstream readers. Adding a new nullable column is safe. Renaming a column is not. When upstream data sources rename a field, add the new column name and deprecate the old one across at least one pipeline run before removing it, so downstream consumers can migrate their column references.

#Frequently Asked Questions

Why does automatic type inference cause problems when migrating Shapefiles to GeoParquet?

Shapefiles infer column types from the first 100 rows of a DBF file. If those rows are all-integer but later rows contain decimals, the inferred type truncates values silently. Explicit pa.schema() definitions lock types before any row is read, preventing downstream query engines from receiving corrupted numeric data.

Should I store geometries as WKT or WKB in GeoParquet?

Always use WKB. WKB is 30–60 % smaller than WKT for equivalent geometry, it is directly readable by spatial indexes without a parsing step, and the GeoParquet specification mandates WKB encoding for the primary geometry column.

How do I handle a source dataset that mixes POLYGON and MULTIPOLYGON rows?

Cast all POLYGON rows to MULTIPOLYGON before writing. This preserves a single declared geometry type in the GeoParquet column metadata, which query engines like DuckDB and Trino require for predicate pushdown on spatial filters to work correctly.

What is the safest strategy for evolving a schema when the upstream source adds or renames columns?

Use pyarrow.unify_schemas() to merge the existing schema with the new source schema. New columns are added with nulls for historical rows; renamed columns should be surfaced by drift detection that compares the current source profile against the manifest’s recorded field list. Never silently drop columns that existing consumers depend on.

#Related

• Handling Null Values in Spatial Schema Mapping
• Preserving Metadata During GeoParquet Conversion
• Building Batch Conversion Pipelines with Python
• Fallback Routing for Failed Migration Jobs
• Shapefile Limitations in Modern Data Stacks

← Back to Data Conversion & Migration Pipelines