#Dead-Letter Queues for Failed Migrations

When a conversion job fails for good — a corrupt source file, an unmappable schema, an error no retry will cure — the right destination is a dead-letter queue: a separate durable queue where the job’s full context lands so it can be triaged, fixed, and replayed rather than lost in a log. The rule that keeps it useful is discipline about the boundary: retries handle transient failures first, and only a job whose retry budget is spent or whose error is classified terminal is routed to the dead-letter queue. The message must carry enough to replay the job without the source system — original payload, error class, attempt count, and a correlation ID — and every arrival should raise an alert so failures are seen, not buried. This page covers the payload shape, redrive and replay, and poison-message handling on SQS and Kafka. It builds directly on Fallback Routing for Failed Migration Jobs, the wider resilience picture for these pipelines.

#Quick-Reference: Dead-Letter Queue Design Choices

#The Failure Boundary and the Envelope

A dead-letter queue is only as good as the boundary you draw before it. Route too eagerly and you fill it with transient blips that a single retry would have cleared; route too late and jobs stall forever in a retry loop that can never succeed. The clean model is two-tier: transient errors — a throttled object-store request, a momentary network partition — are the domain of retry logic for cloud migration pipelines, which handles them automatically inside a bounded budget. Terminal errors — a source file that will never parse, a schema that cannot be mapped, a validation rule that always fails — belong in the dead-letter queue immediately, because no amount of retrying changes the outcome. The classifier that decides “transient versus terminal” is the single most important piece of code in this design, and it should default to terminal for anything it does not recognise, so unknown failures surface rather than spin.

The message envelope is what makes a dead-letter entry actionable. A bare “job failed” record forces an engineer back to the source system to reconstruct what happened; a well-formed envelope lets them triage from the message alone and lets an automated redrive rebuild the exact job. Capture the original job payload, the terminal error class and message, the number of attempts already made, a correlation ID that ties the failure back to its originating batch, and a timestamp. Large inputs — a multi-gigabyte Shapefile — go in by reference as an object-storage URI, never as inline bytes, so the queue message stays small and cheap. This envelope is the same structured record you would emit to structured logging and metrics for migration jobs; the dead-letter queue and the log are two consumers of one well-designed failure event.

Poison messages are the hazard that turns a redrive into an outage. A poison message is one that fails every single time it is processed — so if you replay it back to the main queue unchanged, it fails again, returns to the dead-letter queue, and loops. The defences are an attempt counter that increments on every hop and a hard ceiling past which the message is quarantined rather than redriven, plus a policy that you replay only after the root cause is fixed, never blindly. On SQS this is the maxReceiveCount in the redrive policy backed by an attempt attribute; on Kafka it is an attempt header incremented as the record moves between the main topic and its .DLT companion. Either way, a job that has bounced too many times is a human decision, not an automated retry.

#Implementation: Routing and Redrive on SQS

The handler below classifies the error, sends terminal failures to the dead-letter queue with a full envelope, and the redrive function replays within a hard attempt ceiling.

# Requires: boto3>=1.34, python>=3.11
from __future__ import annotations

import json
import logging
import time
from dataclasses import asdict, dataclass

import boto3

logger = logging.getLogger(__name__)
sqs = boto3.client("sqs")

TERMINAL = (ValueError, KeyError)   # unmappable schema, missing field, etc.
MAX_REDRIVE = 3


class TransientError(Exception):
    """Raised for retryable failures (throttling, network)."""


@dataclass(frozen=True)
class DeadLetter:
    job: dict          # original payload; large inputs stored by URI
    error_class: str
    error_message: str
    attempts: int
    correlation_id: str
    failed_at: float


def route_failure(
    job: dict, exc: Exception, *, attempts: int, correlation_id: str, dlq_url: str
) -> None:
    """Send a terminally-failed job to the dead-letter queue with full context."""
    if isinstance(exc, TransientError):
        raise exc  # let the retry layer handle it; do not dead-letter
    envelope = DeadLetter(
        job=job,
        error_class=type(exc).__name__,
        error_message=str(exc)[:1024],
        attempts=attempts,
        correlation_id=correlation_id,
        failed_at=time.time(),
    )
    sqs.send_message(
        QueueUrl=dlq_url,
        MessageBody=json.dumps(asdict(envelope)),
        MessageAttributes={
            "attempts": {"DataType": "Number", "StringValue": str(attempts)},
            "error_class": {"DataType": "String", "StringValue": envelope.error_class},
        },
    )
    logger.error("Dead-lettered job %s: %s", correlation_id, envelope.error_class)


def redrive(dlq_url: str, main_url: str, *, batch: int = 10) -> int:
    """Replay dead-letter messages to the main queue, capping attempts to
    avoid poison-message loops. Call only after the root cause is fixed."""
    moved = 0
    resp = sqs.receive_message(
        QueueUrl=dlq_url, MaxNumberOfMessages=batch,
        MessageAttributeNames=["All"], WaitTimeSeconds=2,
    )
    for msg in resp.get("Messages", []):
        body = json.loads(msg["Body"])
        if body["attempts"] >= MAX_REDRIVE:
            logger.warning("Quarantining poison message %s", body["correlation_id"])
            continue  # leave in DLQ / move to a quarantine queue
        body["attempts"] += 1
        sqs.send_message(QueueUrl=main_url, MessageBody=json.dumps(body))
        sqs.delete_message(QueueUrl=dlq_url, ReceiptHandle=msg["ReceiptHandle"])
        moved += 1
    logger.info("Redrove %d messages to main queue", moved)
    return moved

#Validation

Verify the routing and the poison guard before trusting them in production. A focused check asserts that terminal errors dead-letter, transient errors do not, and a message at the ceiling is quarantined rather than replayed:

# Requires: pytest>=8.0, moto>=5.0 (mocks SQS)
def test_poison_message_quarantined(dlq_url, main_url, monkeypatch):
    seed_message(dlq_url, attempts=MAX_REDRIVE, correlation_id="poison-1")
    moved = redrive(dlq_url, main_url)
    assert moved == 0                       # not replayed
    assert queue_depth(main_url) == 0       # nothing reached the main queue
    assert queue_depth(dlq_url) == 1        # still parked for a human

Expected behaviour: a ValueError (terminal) produces exactly one dead-letter message with attempts and error_class attributes populated; a TransientError re-raises and never reaches the queue; and a message whose attempts equals MAX_REDRIVE stays in the dead-letter queue with zero moved. In production, watch the dead-letter queue depth as a first-class metric — a healthy pipeline holds this near zero, and any sustained rise is an alert, tracked alongside the run’s other signals in cost and observability for conversion pipelines.

#Edge Cases and Caveats

Oversized payloads. SQS caps a message at 256 KB and Kafka has its own record-size limits. A failed job carrying an inline dataset blows past both. Always store the input by reference — an S3 URI in the envelope — and keep the message to metadata, or the send itself fails and you lose the failure record entirely.

Silent dead-letter accumulation. A dead-letter queue with no alert is a black hole: failures pile up unseen until a downstream consumer notices missing data days later. Wire an alarm to the queue depth from day one, before the first real failure, so the very first arrival pages someone.

Redrive stampedes after an outage. When a widespread transient failure floods the dead-letter queue and you fix the cause, replaying thousands of messages at once can re-overwhelm the same downstream that just recovered. Redrive in throttled batches with the bounded-window discipline used for parallel conversion, not all at once.

#Frequently Asked Questions

What should a dead-letter message payload contain?

Enough to replay the job without consulting the source system: the original job payload, the terminal error class and message, the number of attempts already made, a correlation ID, and a timestamp. Store large inputs by reference — an object-storage URI rather than the bytes — so the queue message stays small. With that envelope, a redrive can reconstruct the exact job and a human can triage it from the message alone.

How is a dead-letter queue different from retry logic?

Retry logic handles transient failures automatically within a job’s retry budget; a dead-letter queue is where a job goes after that budget is exhausted or the error is classified terminal. Retries are for problems that fix themselves — a throttled API, a brief network blip. The dead-letter queue is for problems that need a code change or human decision. They are complementary: retries first, dead-letter queue as the durable backstop.

How do I stop a poison message from looping forever on redrive?

Track an attempt count in the message and refuse to redrive past a hard ceiling, and only replay after the root cause is fixed rather than blindly. A poison message is one that fails every time it is processed; if you redrive it back to the main queue unchanged, it fails again and returns to the dead-letter queue in a loop. Cap redrive attempts, quarantine anything that exceeds the cap, and require a deliberate fix before replay.

#Related

• Fallback Routing for Failed Migration Jobs — the wider resilience and fallback strategy
• Retry Logic for Cloud Migration Pipelines — the transient-failure layer that runs before dead-lettering
• Structured Logging and Metrics for Migration Jobs — emitting the failure event as a structured record
• Cost & Observability for Conversion Pipelines — alerting on dead-letter queue depth
• Parallel Shapefile Conversion with Dask — throttled replay using the same bounded-window discipline

← Back to Fallback Routing for Failed Migration Jobs