#Retry Logic for Cloud Migration Pipelines

Use exponential backoff with full jitter and content-hash idempotency when migrating geospatial data to cloud object storage. For transient failures — 503 SlowDown, 429 Too Many Requests, network timeouts — apply 3–5 retry attempts with a base wait of 4 seconds doubling to a 60-second cap, plus 0–2 seconds of random jitter per attempt. For permanent failures — invalid CRS, unsupported GDAL drivers, corrupt source files — classify them as non-retryable immediately and route them to a fallback routing path rather than consuming the retry budget on jobs that will never succeed.

#Quick-Reference: Error Class to Retry Strategy

#Why Geospatial Pipelines Need Format-Aware Retries

Generic HTTP retry loops break down when migrating spatial datasets. Batch conversion pipelines writing Cloud Optimized GeoTIFF (COG), GeoParquet, or Zarr issue many concurrent multipart uploads, each of which can independently trigger object-storage rate limits. A naive retry that re-executes the full conversion without checking destination state may overwrite a partially written Parquet file, corrupting its row group footer, or produce a duplicate COG overview level.

Idempotency is the foundation of safe retries. Before converting or uploading, compute a SHA-256 hash of the source file and compare it against the src_hash metadata field on the destination object. If they match, the task already succeeded — skip it. This guarantees that restarting a failed batch job does not double-write data or waste compute budget on already-migrated tiles. Pair this with ZSTD compression levels tuned for the target format so that a retried write produces a bit-identical output to the first attempt.

Jitter is equally important. When many pipeline workers fail simultaneously — during a brief storage-gateway hiccup — exponential backoff without jitter causes a synchronized retry spike that overwhelms the same gateway. Adding randomised jitter spreads the load across the backoff window, keeping concurrency inside the provider’s rate limits. AWS explicitly recommends full jitter for distributed systems performing bulk S3 operations.

#Production Implementation

The snippet below uses tenacity (≥ 8.2) for declarative retry control, boto3 (≥ 1.34) for S3 operations, and rasterio (≥ 1.3) for COG writing. It enforces the idempotency check before any conversion, classifies permanent geospatial errors to halt fast, and logs retry sleep intervals for observability.

# tenacity>=8.2, boto3>=1.34, rasterio>=1.3, GDAL>=3.6
import hashlib
import logging
import os
from pathlib import Path

import boto3
import rasterio
from botocore.exceptions import BotoCoreError, ClientError
from rasterio.errors import RasterioIOError
from tenacity import (
    before_sleep_log,
    retry,
    retry_if_exception_type,
    stop_after_attempt,
    wait_exponential,
    wait_random,
)

logger = logging.getLogger("geo_migration.retry")

RETRYABLE = (ClientError, BotoCoreError, RasterioIOError, ConnectionError, TimeoutError)


class PermanentGeoError(Exception):
    """Non-retryable: invalid CRS, unsupported driver, corrupt source."""


def _source_hash(path: str) -> str:
    return hashlib.sha256(Path(path).read_bytes()).hexdigest()


def _already_migrated(s3, bucket: str, key: str, src_hash: str) -> bool:
    try:
        head = s3.head_object(Bucket=bucket, Key=key)
        return head.get("Metadata", {}).get("src_hash") == src_hash
    except ClientError as exc:
        if exc.response["Error"]["Code"] == "404":
            return False
        raise


@retry(
    stop=stop_after_attempt(5),
    wait=wait_exponential(multiplier=2, min=4, max=60) + wait_random(0, 2),
    retry=retry_if_exception_type(RETRYABLE),
    before_sleep=before_sleep_log(logger, logging.WARNING),
    reraise=True,
)
def migrate_raster_to_cog(
    src_path: str,
    dest_bucket: str,
    dest_key: str,
    region: str = "us-east-1",
) -> dict:
    """Convert a local raster to COG and upload with idempotency tracking."""
    src_hash = _source_hash(src_path)
    s3 = boto3.client("s3", region_name=region)

    if _already_migrated(s3, dest_bucket, dest_key, src_hash):
        logger.info("Skipping %s — already migrated.", dest_key)
        return {"status": "skipped", "key": dest_key}

    tmp = f"/tmp/{Path(dest_key).name}"
    try:
        with rasterio.open(src_path) as src:
            if not src.crs or not src.crs.is_valid:
                raise PermanentGeoError(f"Invalid CRS in {src_path}")
            profile = {**src.profile, "driver": "COG", "compress": "deflate",
                       "tiled": True, "blockxsize": 256, "blockysize": 256}
            with rasterio.open(tmp, "w", **profile) as dst:
                dst.write(src.read())

        s3.upload_file(tmp, dest_bucket, dest_key,
                       ExtraArgs={"Metadata": {"src_hash": src_hash}})
        logger.info("Migrated %s → s3://%s/%s", src_path, dest_bucket, dest_key)
        return {"status": "success", "key": dest_key}

    except ClientError as exc:
        code = exc.response["Error"]["Code"]
        if code in ("AccessDenied", "InvalidParameter"):
            raise PermanentGeoError(f"Permanent S3 error {code}") from exc
        raise
    except RasterioIOError as exc:
        if "CRS" in str(exc) or "not recognized" in str(exc):
            raise PermanentGeoError(str(exc)) from exc
        raise
    finally:
        if os.path.exists(tmp):
            os.remove(tmp)

The @retry decorator catches only RETRYABLE exceptions; PermanentGeoError propagates immediately without consuming retry budget. The combined wait_exponential + wait_random adds full jitter on top of the exponential floor, spreading concurrent worker retries across the backoff window without a synchronised spike against the storage gateway.

#Validation and Verification

After migration, confirm output integrity before marking a batch complete.

COG validation via rio-cogeo:

# pip install rio-cogeo>=3.6
rio cogeo validate s3://my-bucket/output/tile.tif
# Expected: tile.tif is a valid cloud optimized GeoTIFF

GeoParquet read smoke-test:

import geopandas as gpd  # geopandas>=0.14, pyarrow>=14
gdf = gpd.read_parquet("s3://my-bucket/output/parcels.parquet")
assert len(gdf) > 0 and gdf.crs is not None, "Output invalid"

Pipeline-level metrics to track:

#Edge Cases and Caveats

Very large rasters (> 1 GB per file). SHA-256 hashing the entire source before conversion adds several seconds of local disk read. For files above 1 GB, hash only the first and last 8 MB plus the file size to form a fast fingerprint, and store the full hash as a background verification step after upload. This keeps the idempotency gate cheap without losing correctness for normal-size files.

Mixed CRS sources in a single batch. GDAL reprojection failures during multi-source mosaics can surface as RasterioIOError with no CRS mention in the message, causing a retry of what is actually a permanent configuration error. Add a pre-flight CRS validation step — using schema mapping conventions to normalise coordinate reference systems — before the retry-decorated function to catch these early and avoid burning your retry budget on unresolvable jobs.

Streaming vs. batch orchestrators. In streaming pipelines (Kafka + Faust, or Kinesis + Lambda), tenacity’s in-process retry loop blocks the consumer thread. Prefer a short stop_after_attempt(2) inside the consumer and rely on the queue’s own redelivery mechanism (visibility timeout or retry topic) for subsequent attempts. Reserve the full 5-attempt loop for batch orchestrators like Airflow or Prefect where blocking a worker task is acceptable.

#Frequently Asked Questions

How many retry attempts should a cloud migration pipeline use?

Three to five attempts resolves the vast majority of transient throttling from cloud object storage APIs. Exceeding five attempts typically signals a systemic issue — network partition, IAM policy error, or storage quota — that retries alone cannot fix. Set stop_after_attempt(5) and route failures to a dead-letter queue after that.

Why does idempotency matter for GeoParquet and COG migrations?

A retried upload that overwrites a partially written GeoParquet file can corrupt row group footers, making the file unreadable by DuckDB or Spark. Checking the SHA-256 hash stored in object metadata before each attempt ensures the operation is skipped when the destination already matches the source, preventing data corruption and wasted compute. This is especially important when row group sizing produces large footers that take time to write atomically.

What is the difference between full jitter and equal jitter for migration retries?

Full jitter randomises the entire wait interval between zero and the computed cap, giving maximum spread across concurrent workers. Equal jitter preserves half the backoff floor, reducing the chance of very short waits during high concurrency. For bulk spatial migrations with many parallel workers, full jitter — wait_random combined with wait_exponential in tenacity — generally reduces storage-gateway contention more effectively.

#Related

• Fallback Routing for Failed Migration Jobs — dead-letter queues, quarantine buckets, and recovery path design
• Building Batch Conversion Pipelines with Python — parallelism patterns for large geospatial batches
• Preserving Metadata During GeoParquet Conversion — keeping CRS and attribute metadata intact across format conversions
• Schema Mapping for Legacy to Modern Formats — normalising CRS and field types before migration
• ZSTD Compression Levels for Geospatial Data — choosing compression settings that produce deterministic, idempotent outputs

← Back to Fallback Routing for Failed Migration Jobs