#Incremental Conversion and Change Data Capture

Most geospatial conversion pipelines are written once, against an empty target, and then run forever as though the target were still empty. Every night the job reads the whole source estate, converts every feature, and replaces the dataset. It works, it is easy to reason about, and it stays affordable right up until the dataset gets big — at which point the nightly window stops closing and somebody is asked to make it faster.

The insight that fixes it is unglamorous: the cost of a full rebuild is proportional to the dataset, and the value of a run is proportional to the change. A national address estate of 34 million features where 0.4% of records move each day is spending 99.6% of its compute reproducing bytes that already exist. Incremental conversion inverts that ratio, and the price is a small amount of bookkeeping: a stable key, a fingerprint, a manifest, and a rule for turning changed features into rewritten files.

This page sits under Data Conversion & Migration Pipelines and assumes the machinery that section describes is already working — a batch conversion pipeline that produces valid GeoParquet, schema mapping that survives source drift, and metadata preservation that keeps CRS intact. What follows is how to run it on the delta rather than the whole.

#Prerequisites

• Python 3.10+ with geopandas>=1.0, pyarrow>=16, shapely>=2.0, and duckdb>=1.0 for manifest joins at scale
• A stable feature identifier in the source, or a documented rule for deriving one. This is the hard prerequisite; everything else is mechanics.
• A partitioned target layout — Hive-style directories, spatial or temporal — so a change can be localised to a small number of files
• Object storage with atomic single-object writes, which every major provider offers, so the manifest swap is a single durable operation
• A retention policy for superseded files, since incremental rewriting leaves orphans that nothing will clean up on its own

#What Change Detection Actually Requires

A change-detection system answers one question per feature: is this the same as last time? Answering it reliably requires three things, and the failure of any one of them silently degrades the pipeline back into a full rebuild.

A key that survives editing. If the source assigns a new identifier whenever a feature is edited, every edit looks like a delete plus an insert. That is tolerable. If it reassigns identifiers on every export — which row-order-derived keys do — then every run looks like a complete replacement and no amount of hashing helps. Establishing the key is the first real task, and sometimes it means persuading the upstream system to emit one.

A fingerprint that is stable under harmless variation. The same parcel exported twice may arrive with rings in a different orientation, coordinates that differ in the sixteenth decimal, or attributes in a different column order. All three change the bytes without changing the feature. The fingerprint must normalise them away, which is why it is computed over canonicalised geometry — at the same precision the storage layer uses, as described in geometry encoding and coordinate precision — and over a sorted, explicitly typed attribute tuple.

A record of where each feature lives. Knowing that a feature changed is not enough; you must know which file to rewrite. The manifest therefore stores the partition alongside the fingerprint, which is also what makes cross-partition moves detectable.

#The Workflow

#1. Pin down the key

Interrogate the source. A UPRN, a parcel reference, a permanent asset tag — anything the upstream system treats as durable. If none exists, derive one from immutable attributes and record the derivation rule in the manifest, so a later change to the rule is a visible schema event rather than a mysterious mass update. Never derive a key from row order, file name, or geometry alone.

#2. Normalise, then fingerprint

Round coordinates to the storage grid, orient rings canonically, sort attribute names, render nulls explicitly, and hash the result. The output is a fixed-width digest per feature. Crucially, fingerprint after normalisation and before any partitioning decision, so the digest describes the feature rather than its placement.

#3. Diff against the previous manifest

An outer join on the key yields four classes. Rows present in both with equal digests and equal partitions are untouched. Equal digests but different partitions mean the feature moved — a rewrite of two files, not one. Different digests mean changed. Present only in the source means added; present only in the manifest means removed.

#4. Resolve to the smallest file set

Collect the union of partitions implicated by added, changed, removed, and moved features. That set — typically a small fraction of the estate — is what gets rewritten. Every other file is carried forward by reference in the new manifest, untouched and still cached downstream.

#5. Write beside, then swap

Write each rewritten partition under a new object key. When every write has succeeded, write the new manifest and flip the dataset pointer in one operation. If any write fails, the pointer never moves, the old manifest still describes a complete and valid dataset, and the failed run costs nothing but orphaned objects a retention job will collect. This is the same discipline the fallback routing work applies to individual jobs, raised to the level of the dataset.

#Production Implementation

The fingerprint-and-diff core is small. The care is all in normalisation, because that is where false positives come from — and a false positive rate of even 5% turns a 2% rewrite into a 7% one.

# Requires: geopandas>=1.0, shapely>=2.0, pyarrow>=16  (Python 3.10+)
from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass
from typing import Iterable

import geopandas as gpd
import pyarrow as pa
import pyarrow.parquet as pq
from shapely import normalize, set_precision, to_wkb


@dataclass(frozen=True)
class Delta:
    added: set[str]
    changed: set[str]
    removed: set[str]
    moved: set[str]

    @property
    def touched(self) -> set[str]:
        return self.added | self.changed | self.removed | self.moved

    def summary(self) -> dict[str, int]:
        return {
            "added": len(self.added), "changed": len(self.changed),
            "removed": len(self.removed), "moved": len(self.moved),
        }


def fingerprint_frame(
    frame: gpd.GeoDataFrame,
    key_column: str,
    *,
    grid: float,
    attributes: tuple[str, ...],
) -> dict[str, str]:
    """Map every feature key to a digest that is stable under harmless re-export.

    Normalisation is the whole point: without it, a re-export with flipped ring
    orientation or a different column order marks the entire dataset changed.
    """
    if key_column not in frame.columns:
        raise KeyError(f"key column {key_column!r} missing from the source")
    if frame[key_column].isna().any():
        raise ValueError("null feature keys — change detection cannot proceed")
    if frame[key_column].duplicated().any():
        dupes = int(frame[key_column].duplicated().sum())
        raise ValueError(f"{dupes} duplicate feature keys — the key is not stable")

    # Canonical geometry: snapped to the storage grid, rings in a fixed
    # orientation and vertex order, serialised as WKB.
    geoms = normalize(set_precision(frame.geometry.to_numpy(), grid_size=grid))

    ordered = tuple(sorted(attributes))
    out: dict[str, str] = {}
    for key, geom, row in zip(frame[key_column], geoms, frame[list(ordered)].itertuples(index=False)):
        payload = {
            "g": to_wkb(geom, hex=True) if geom is not None else None,
            # Explicit null rendering and a fixed field order keep the digest
            # independent of the source's column ordering.
            "a": {name: (None if value != value else value)
                  for name, value in zip(ordered, row)},
        }
        blob = json.dumps(payload, sort_keys=True, default=str, separators=(",", ":"))
        out[str(key)] = hashlib.blake2b(blob.encode("utf-8"), digest_size=16).hexdigest()
    return out


def diff_against_manifest(
    current: dict[str, str],
    current_partition: dict[str, str],
    previous: dict[str, tuple[str, str]],
) -> Delta:
    """Classify every key. previous maps key -> (digest, partition)."""
    added, changed, moved = set(), set(), set()
    for key, digest in current.items():
        prior = previous.get(key)
        if prior is None:
            added.add(key)
            continue
        prior_digest, prior_partition = prior
        if prior_digest != digest:
            changed.add(key)
        elif prior_partition != current_partition.get(key):
            # Same content, different home: both files must be rewritten or a
            # duplicate is left behind in the old partition.
            moved.add(key)
    removed = set(previous) - set(current)
    return Delta(added=added, changed=changed, removed=removed, moved=moved)


def partitions_to_rewrite(
    delta: Delta,
    current_partition: dict[str, str],
    previous: dict[str, tuple[str, str]],
) -> set[str]:
    """Every partition implicated on either side of the change."""
    affected: set[str] = set()
    for key in delta.added | delta.changed | delta.moved:
        part = current_partition.get(key)
        if part:
            affected.add(part)
    for key in delta.removed | delta.changed | delta.moved:
        prior = previous.get(key)
        if prior:
            affected.add(prior[1])
    return affected

The commit is deliberately boring, which is the point:

# Requires: boto3>=1.34 — write beside, then flip one pointer
import json

import boto3
from botocore.exceptions import BotoCoreError, ClientError


def commit_manifest(bucket: str, dataset: str, manifest: dict, version: int) -> None:
    """Publish a new manifest and move the pointer in a single durable write."""
    client = boto3.client("s3")
    manifest_key = f"{dataset}/_manifests/v{version:06d}.json"
    try:
        client.put_object(
            Bucket=bucket, Key=manifest_key,
            Body=json.dumps(manifest, sort_keys=True).encode("utf-8"),
            ContentType="application/json",
        )
        # The pointer is the transaction boundary. Until this PUT lands, every
        # reader still sees the previous complete dataset.
        client.put_object(
            Bucket=bucket, Key=f"{dataset}/_current.json",
            Body=json.dumps({"manifest": manifest_key, "version": version}).encode("utf-8"),
            ContentType="application/json",
            CacheControl="no-cache",
        )
    except (BotoCoreError, ClientError) as exc:
        raise RuntimeError(f"manifest commit failed at v{version}: {exc}") from exc

#Reference Matrix

Measured on a 34.1 million-feature national address estate partitioned into 8,900 GeoParquet files by administrative area, with a typical daily change rate of 0.4%.

The third row is the target for most estates. The fourth is faster still but only applies where features are genuinely immutable — sensor readings, transaction events, observation records — because an append-only log cannot express an edit. The fifth costs slightly more than the third and buys transactional guarantees; take that trade when more than one process writes, and see Iceberg tables with GeoParquet for the query-side implications.

#Failure Modes and Gotchas

A key that is not actually stable. The pipeline runs, reports that 100% of features changed, and quietly performs a full rebuild at higher cost than the full rebuild it replaced. Guard against it explicitly: if the changed fraction exceeds a threshold — 20% is a reasonable alarm — fail the run and require a human to confirm, rather than silently rewriting everything.

Fingerprinting before normalisation. The single most common cause of the above. A source that exports rings clockwise on Mondays and counter-clockwise on Tuesdays will produce a 100% change rate every other day, and the diagnosis is not obvious from the logs. Normalise geometry and attributes first, always.

Forgetting the move case. Upserting a feature into its new partition without deleting it from the old leaves a duplicate that only shows up in spatial queries returning two results for one address. The implementation above tracks the previous partition specifically to catch this.

Orphaned objects accumulating forever. Every incremental run supersedes files without deleting them, which is exactly what makes rollback possible — and what makes storage grow without bound. Run a retention job that deletes objects unreferenced by any manifest older than the rollback window, and monitor its cost the way cost-per-GB tracking monitors the pipeline’s.

Layout drift under repeated partial rewrite. Rewriting one partition at a time means new files no longer share the global sort order the original bulk load established, so space-filling-curve locality degrades and bbox skipping gets steadily worse. Schedule a periodic full re-sort — monthly is usually enough — and treat the incremental path as the fast path, not the only path.

#Frequently Asked Questions

Why not just re-convert the whole dataset every night?

Because the cost grows with the size of the dataset while the value grows with the size of the change. A 900 GB estate where 0.4% of features move each day burns compute, write requests, and a maintenance window to reproduce 99.6% of the same bytes. It also destroys the physical layout: a full rewrite re-sorts and re-partitions everything, invalidating every cached read and forcing consumers to re-download data that did not change.

What makes a good feature fingerprint for change detection?

A hash over the normalized geometry plus the sorted attribute tuple. Normalization is what makes it reliable: round coordinates to the storage precision, put rings in a canonical orientation, and serialise attributes in a fixed field order with explicit null handling. Without normalization the same feature hashes differently after a harmless re-export, and the pipeline decides that every feature changed.

How do I handle a feature that moves from one spatial partition to another?

Treat it as a delete in the old partition and an insert in the new one, and rewrite both. This is the case that a naive upsert gets wrong: writing the feature into its new partition without removing it from the old leaves a duplicate that spatial queries will return twice. The diff must therefore compare each feature’s current partition against the partition recorded in the previous manifest, not only its content.

Do I need a table format like Iceberg or Delta to do incremental updates?

No, but a table format removes work you would otherwise write yourself. Its value is transactional file-list swapping, snapshot isolation for readers, and time travel — all of which you can approximate with a manifest object and atomic pointer swap. Adopt one when several writers touch the same dataset or when readers need consistent snapshots during a rewrite; a manifest is sufficient when a single pipeline owns the dataset.

#Related

• Partition Overwrite vs Append for Spatial Tables — choosing a write mode, and what each one costs readers
• Detecting Changed Features with Geometry Hashing — the normalisation rules that keep a digest stable
• Building Batch Conversion Pipelines with Python — the full-conversion path this one accelerates
• Fallback Routing for Failed Migration Jobs — the per-job version of the commit discipline used here
• Cost and Observability for Conversion Pipelines — measuring whether the incremental path is actually paying off

← Back to Data Conversion & Migration Pipelines