Data Compression
  1. Home
  2. Data Conversion & Migration Pipelines for Cloud-Native Geospatial Storage
  3. Preserving Metadata During GeoParquet Conversion
  4. Round-Tripping GeoParquet Back to Shapefile
Section: Data Conversion & Migration Pipelines for Cloud-Native Geospatial Storage 15 min read

Round-Tripping GeoParquet Back to Shapefile

Exporting to shapefile is a lossy operation that reports no errors, so the only way to do it responsibly is to enumerate the losses in advance and ship the enumeration with the data. Some consumers genuinely still require shapefiles — a regulator’s submission portal, a decades-old desktop workflow — and refusing to produce one is rarely an option. What is an option is making the trip reversible. This page extends preserving metadata during GeoParquet conversion.

Quick Reference

What is lost Mitigation Reversible? Primary Use Case
Field names over 10 chars Explicit name mapping in a sidecar Yes, with the sidecar Every export
Mixed geometry types One shapefile per type Yes, by convention Multi-type layers
int64, timestamps, booleans Declare original types in the sidecar Yes, with the sidecar Typed attribute schemas
GeoParquet geo metadata Ship the JSON alongside Yes, with the sidecar CRS and covering columns
Text encoding Write a .cpg Yes Non-English attributes
2 GB per component Split into numbered parts Yes, by convention Large national layers

What the Format Cannot Carry

The shapefile is three or more files that must travel together, and each carries a different limitation. The .dbf holds attributes in a 1980s dBASE layout with ten-character names and a small set of types. The .shp holds geometry of exactly one type, with a 32-bit byte offset in its index that caps each component at 2 GB. Nothing anywhere holds structured metadata.

Every one of these is a silent limitation. The writer truncates the name, coerces the type, and omits the metadata, then reports success. A recipient opening the result sees a valid layer that looks complete — which is precisely why the loss list has to be produced by the exporter rather than discovered by the consumer.

The types deserve specific attention because the coercions are subtle. A 64-bit identifier becomes a double and silently loses precision above 2^53. A timestamp with a time zone becomes a date, dropping the time and the zone. A boolean becomes a one-character T/F field. Each is individually recoverable if you know what it was, and unrecoverable if you do not.

What each part of the schema becomes on the way into a shapefile A source schema on the left lists a long field name, a 64-bit identifier, a timestamp with time zone, a boolean, a mixed geometry column, and a GeoParquet geo metadata block. Arrows lead to what each becomes on the right: a ten-character truncated name, a double that loses precision above two to the fifty-third, a date with the time and zone dropped, a one-character text field, one file per geometry type, and nothing at all for the metadata. A note records that none of these transformations reports an error. GeoParquet schema What the shapefile holds parcel_reference_number PARCEL_REF · rest discarded uprn: int64 double · exact only below 2⁵³ surveyed_at: timestamp(tz) date · time and zone dropped is_active: bool one-character T / F text field geometry: Polygon + Point two separate shapefiles geo metadata (CRS, covering) nothing — a .prj is the only survivor Every row above is applied silently: the export reports success and the layer opens correctly.

Implementation

python
# Requires: geopandas>=1.0, pyogrio>=0.9, pyarrow>=16  (Python 3.10+)
from __future__ import annotations

import json
from dataclasses import dataclass, field
from pathlib import Path

import geopandas as gpd
import pyarrow.parquet as pq

MAX_COMPONENT_BYTES = 2 * 1024**3 - (64 << 20)     # stop short of the hard 2 GB ceiling


@dataclass
class LossReport:
    truncated_names: dict[str, str] = field(default_factory=dict)
    coerced_types: dict[str, str] = field(default_factory=dict)
    geometry_split: list[str] = field(default_factory=list)
    dropped_metadata: list[str] = field(default_factory=list)
    parts: int = 1

    def to_json(self) -> str:
        return json.dumps({
            "field_names": self.truncated_names,
            "original_types": self.coerced_types,
            "geometry_types": self.geometry_split,
            "dropped_metadata": self.dropped_metadata,
            "parts": self.parts,
            "note": "apply field_names and original_types in reverse to restore the schema",
        }, indent=2, sort_keys=True)


def plan_names(columns: list[str]) -> dict[str, str]:
    """Deterministic ten-character names, refusing ambiguity.

    Letting the writer truncate means collisions are resolved by column order,
    so two exports of the same layer can disagree about which field is which.
    """
    mapping: dict[str, str] = {}
    used: set[str] = set()
    for name in columns:
        short = name[:10].upper()
        if short in used:
            raise ValueError(
                f"{name!r} truncates to {short!r}, which is already taken — "
                f"supply an explicit short name for one of them"
            )
        used.add(short)
        mapping[name] = short
    return mapping


def export(
    source: Path, out_dir: Path, stem: str, *, names: dict[str, str] | None = None
) -> LossReport:
    """Export to shapefile and produce the report that makes it reversible."""
    frame = gpd.read_parquet(source)
    if frame.empty:
        raise ValueError(f"{source} has no features")
    if frame.crs is None:
        raise ValueError("refusing to export without a CRS — the .prj would be absent")

    out_dir.mkdir(parents=True, exist_ok=True)
    report = LossReport()

    attributes = [c for c in frame.columns if c != frame.geometry.name]
    report.truncated_names = names or plan_names(attributes)
    for column in attributes:
        dtype = str(frame[column].dtype)
        if dtype.startswith(("int64", "datetime64", "bool")):
            report.coerced_types[column] = dtype

    geo = pq.read_schema(source).metadata or {}
    if b"geo" in geo:
        report.dropped_metadata.append("geo")

    for geom_type, subset in frame.groupby(frame.geometry.geom_type):
        report.geometry_split.append(str(geom_type))
        renamed = subset.rename(columns=report.truncated_names)
        target = out_dir / f"{stem}_{str(geom_type).lower()}.shp"
        renamed.to_file(target, driver="ESRI Shapefile", encoding="utf-8")
        # The writer emits .prj from the CRS; .cpg has to be written explicitly
        # or the recipient is left guessing the code page.
        target.with_suffix(".cpg").write_text("UTF-8", encoding="ascii")

        if target.stat().st_size > MAX_COMPONENT_BYTES:
            raise RuntimeError(
                f"{target.name} is within reach of the 2 GB component ceiling — "
                f"split the source before exporting"
            )

    (out_dir / f"{stem}_loss_report.json").write_text(report.to_json(), encoding="utf-8")
    return report

Validation

Verify the trip by reversing it and comparing against the source, using the report. Anything the report cannot restore is a genuine loss and should be named in the delivery note.

python
# Requires: geopandas>=1.0 — reverse the trip and diff against the source
import json
import geopandas as gpd

report = json.loads(open("out/parcels_loss_report.json", encoding="utf-8").read())
inverse = {short: long for long, short in report["field_names"].items()}

restored = gpd.read_file("out/parcels_polygon.shp").rename(columns=inverse)
for column, original in report["original_types"].items():
    restored[column] = restored[column].astype(original)

source = gpd.read_parquet("parcels.parquet")
assert set(restored.columns) == set(source.columns), "column set did not survive"
assert len(restored) == len(source[source.geometry.geom_type == "Polygon"]), "feature loss"
print("round trip restored the schema; geometry precision is the remaining difference")

Expected results: identical column names and types after applying the report, identical feature counts per geometry type, and coordinates matching to the precision the shapefile stored. Any 64-bit identifier above 2^53 will differ — check for that explicitly rather than assuming it did not occur.

The sidecar is what turns a lossy export into a reversible protocol Two round trips of the same data. In the first, GeoParquet is exported to shapefile and read back with no accompanying information: the recipient gets truncated names, coerced types, and no CRS metadata beyond the projection file, and cannot restore the original schema. In the second, a loss report travels with the shapefile: the recipient applies the field name mapping and the original type list in reverse and recovers the schema exactly, leaving only the geometry precision the shapefile stored. The same export, with and without its sidecar Files only GeoParquet .shp .dbf .prj PARCEL_REF, doubles, no types the schema cannot be restored — the information needed is not in the files Files + report GeoParquet .shp .dbf .prj .cpg + loss_report.json schema restored exactly only geometry precision remains lost, and the report states what it was The report costs a few kilobytes and is the difference between a delivery and a data-loss incident. Where a DBF text width should come from Two ways of declaring a text field width in a DBF. Taking the width from a schema default truncates every value longer than the default, silently and per row, so a small number of long descriptions lose their tails. Measuring the maximum length in the data and declaring that width preserves every value, at the cost of a wider fixed-width field for every row including the short ones. Width from a schema default (80) stored truncated, per row, silently 2,418 of 4.2 M values lose their tails — and nothing anywhere records that they did Width from the measured maximum (214) every value stored in full The field is wider for every row, which costs bytes — and loses nothing, which is the point.

Edge Cases and Caveats

Attributes whose values exceed the DBF field width. A text field is declared with a fixed width, and a value longer than it is truncated per row, not per schema. That is a silent per-record loss that no schema comparison detects. Measure the maximum length of every text column before exporting and declare widths from the data.

The 2 GB ceiling arriving mid-write. The .shp and .dbf each have a 32-bit offset limit, and hitting it produces a file that appears written and reads back corrupt past the boundary. Split before you approach it — the implementation stops well short — rather than discovering it in a consumer’s desktop tool.

A CRS with no shapefile-compatible representation. .prj holds a WKT string, and some modern CRS definitions, particularly those with time-dependent transformations, have no faithful WKT1 form. Export the closest match, record the exact original definition in the report, and say so in the delivery note — the same discipline as preserving CRS metadata in the forward direction.

Frequently Asked Questions

What is actually lost exporting GeoParquet to shapefile?

Field names beyond ten characters, any mixture of geometry types in one file, most type precision — 64-bit integers, timestamps with time zones, and booleans all degrade — and every piece of structured metadata the GeoParquet carried, including the geo block, the covering columns, and any custom key-value pairs. The 2 GB per-component ceiling also caps how much data can travel in one file. None of these produce an error.

Can a GeoParquet to shapefile to GeoParquet round trip be lossless?

Not through the shapefile alone, because the information needed to reverse the trip is not in the files. It can be lossless if you ship a sidecar mapping alongside — full field names, original types, the CRS definition, and the geometry-type split — and the recipient applies it on the way back. That makes the round trip a documented protocol rather than a lossy conversion, which is the best available outcome.

Should I let the writer truncate field names automatically?

No. Automatic truncation appends digits on collision, and which colliding field gets which digit depends on column order, so two exports of the same layer can assign them differently. Supply an explicit mapping so the short names are stable, meaningful, and reversible, and fail the export if two long names would collide rather than letting the writer choose.

← Back to Preserving Metadata During GeoParquet Conversion