#CI/CD Validation Hooks for GeoParquet Conversion

A conversion pipeline that writes GeoParquet without a gate is one silent schema drift away from shipping corrupt data to every downstream query engine. The fix is to treat converted output like code: run a pytest validator on every pull request that asserts four things against a pinned contract — the Arrow schema matches, the CRS is exactly the target CRS, every geometry is valid, and the row count is within expected bounds — and make that job a required status check so a failing conversion physically cannot merge. This page gives you the validate() function, the pytest wrapper, and the GitHub Actions workflow to wire it together. It extends the conversion logic in Building Batch Conversion Pipelines with Python with the safety net that keeps a green pipeline honest.

#Quick-Reference: The CI Gate Checklist

#Why These Checks, and Where They Run

The value of a CI gate is proportional to how early and how specifically it fails. A schema check is the highest-leverage of the set because a single retyped column — an int32 parcel ID that became a string, or a float64 area that lost precision — breaks predicate pushdown and joins downstream without raising any error at write time. Reading the Parquet footer and comparing the Arrow schema field-by-field against a pinned contract catches this in milliseconds. Pair it with the row-count bound, which reads the same footer’s num_rows, and you have caught both structural drift and silent truncation before decoding a single geometry. These footer-only checks are cheap enough to run on every file in the batch on every pull request.

CRS identity deserves its own gate because CRS loss is both common and invisible. A converter that drops the geo metadata, or writes coordinates in one CRS while labelling them another, produces a file that opens fine and plots wrong. The check reads the GeoParquet geo metadata key, confirms it is present and spec-compliant, and asserts the recorded CRS equals the target — the same discipline covered in depth in preserving CRS metadata in GeoParquet. Because it too reads only metadata, it belongs in the per-file, every-PR tier.

Geometry validity is the one expensive check, since it must decode and test every geometry with an is_valid predicate. Running it on a full national dataset in CI would blow your job time budget, so scope it: validate only the files changed in the pull request, plus a fixed random sample of the rest, and backstop that with a nightly full scan. When invalid geometries do appear, the same null- and validity-handling policy from handling null values in spatial schema mapping decides whether the gate should fail the build or quarantine the offending rows.

#The validate() Function and pytest Wrapper

The validator reads only what each check needs — the footer for cheap checks, a full read for geometry validity — and returns a structured report. The pytest layer parametrises over files so a failure names the file and the broken rule.

# Requires: pyarrow>=14.0, geopandas>=0.14, shapely>=2.0, pyproj>=3.6, pytest>=8.0
from __future__ import annotations

import json
from dataclasses import dataclass, field

import geopandas as gpd
import pyarrow.parquet as pq
from pyproj import CRS


@dataclass
class ValidationReport:
    path: str
    failures: list[str] = field(default_factory=list)

    @property
    def ok(self) -> bool:
        return not self.failures


def validate(
    path: str,
    *,
    expected_schema: dict[str, str],
    target_epsg: int,
    min_rows: int,
    check_geometry: bool = False,
) -> ValidationReport:
    """Validate one GeoParquet file against a pinned conversion contract."""
    report = ValidationReport(path=path)
    pf = pq.ParquetFile(path)
    arrow_schema = pf.schema_arrow

    # 1. Schema equality (cheap: footer only)
    actual = {f.name: str(f.type) for f in arrow_schema}
    if actual != expected_schema:
        report.failures.append(f"schema mismatch: {actual} != {expected_schema}")

    # 2. Row-count bound (cheap: footer only)
    if pf.metadata.num_rows < min_rows:
        report.failures.append(
            f"row count {pf.metadata.num_rows} below minimum {min_rows}"
        )

    # 3. CRS identity + GeoParquet spec compliance (cheap: geo metadata)
    meta = arrow_schema.metadata or {}
    geo_raw = meta.get(b"geo")
    if geo_raw is None:
        report.failures.append("missing GeoParquet 'geo' metadata key")
    else:
        geo = json.loads(geo_raw)
        primary = geo["primary_column"]
        crs_obj = geo["columns"][primary].get("crs")
        if crs_obj is None or CRS.from_user_input(crs_obj) != CRS.from_epsg(target_epsg):
            report.failures.append(f"CRS is not EPSG:{target_epsg}")

    # 4. Geometry validity (expensive: full decode — sample or changed files only)
    if check_geometry:
        gdf = gpd.read_parquet(path)
        n_invalid = int((~gdf.geometry.is_valid).sum())
        n_empty = int(gdf.geometry.is_empty.sum())
        if n_invalid or n_empty:
            report.failures.append(
                f"{n_invalid} invalid and {n_empty} empty geometries"
            )

    return report

The pytest wrapper turns each converted file into a test case:

# tests/test_geoparquet_gate.py  — pytest>=8.0
import glob
import pytest
from validators import validate  # the module above

EXPECTED = {"parcel_id": "int64", "area_m2": "double", "geometry": "binary"}
FILES = sorted(glob.glob("build/geoparquet/*.parquet"))

@pytest.mark.parametrize("path", FILES)
def test_conversion_contract(path: str) -> None:
    report = validate(
        path, expected_schema=EXPECTED, target_epsg=4326,
        min_rows=1, check_geometry=True,
    )
    assert report.ok, f"{path}: " + "; ".join(report.failures)

#Validation: The GitHub Actions Gate

Run the suite on pull requests and mark the job a required status check so a red gate blocks merge:

# .github/workflows/geoparquet-gate.yml
name: geoparquet-gate
on: [pull_request]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install pyarrow>=14.0 geopandas>=0.14 shapely>=2.0 pyproj>=3.6 pytest>=8.0
      - run: python -m pytest tests/test_geoparquet_gate.py -q

Expected behaviour: a clean conversion prints one passing test per file and exits 0; a drifted schema or lost CRS fails the specific parametrised case with the offending file path and rule. Once the validate job is marked required in branch protection, GitHub disables the merge button until it is green. For failures that recur across runs rather than blocking a single PR, feed the reports into cost and observability for conversion pipelines so you can see whether a source system is regressing over time.

#Edge Cases and Caveats

Schema equality is too strict for additive changes. A byte-for-byte schema comparison fails the moment someone adds a legitimate new column. Version your contract and allow a superset check for additive-only migrations, reserving strict equality for the columns downstream consumers actually depend on.

Geometry validity at national scale blows the CI budget. Decoding every geometry in a multi-million-row dataset can take minutes. Keep check_geometry=True scoped to changed or sampled files in the per-PR job, and move the exhaustive scan to a scheduled nightly workflow that files an issue rather than blocking a human.

CRS equality via string comparison gives false negatives. "EPSG:4326", an authority code, and a full WKT2 string can all describe the same CRS. Always compare with pyproj.CRS equality rather than string matching, or an identical CRS expressed two ways will fail the gate.

#Frequently Asked Questions

What is the minimum set of checks a GeoParquet CI gate should run?

Four checks catch the overwhelming majority of conversion regressions: schema equality against a pinned contract, CRS identity so no reprojection or CRS loss slipped in, geometry validity so no self-intersecting or empty geometries survive, and a row-count bound so silent truncation is caught. Add GeoParquet spec compliance — a present and well-formed geo metadata key — as a fifth gate when you publish to external consumers.

Should validation run on every file or a sample in CI?

Run the cheap footer-only checks — schema, CRS, row count, and geo metadata presence — on every file, because they read only the Parquet metadata and cost milliseconds. Run the expensive geometry-validity scan on a representative sample or on files changed in the pull request, since it must decode every geometry. A nightly full scan backstops the per-PR sample.

How do I make the check a required status before merge?

Run the pytest suite as a GitHub Actions job on the pull_request event, then mark that job a required status check in the branch protection rules for your default branch. Once required, GitHub blocks the merge button until the job reports success, so a conversion that fails schema, CRS, geometry, or row-count validation cannot reach the main branch.

#Related

• Building Batch Conversion Pipelines with Python — the conversion pipeline this gate protects
• Parallel Shapefile Conversion with Dask — the parallel job whose outputs feed the gate
• Preserving CRS Metadata in GeoParquet — the CRS contract the gate enforces
• Handling Null Values in Spatial Schema Mapping — policy for invalid and null geometries
• Cost & Observability for Conversion Pipelines — tracking recurring gate failures over time

← Back to Building Batch Conversion Pipelines with Python