#Tiered Storage for Large Spatial Datasets

For a large spatial estate, the right default is gold serving layers hot in standard storage, silver analytical layers in Intelligent-Tiering or infrequent-access, and bronze raw layers in archive once reprocessing stops. This mapping follows the economics: the bronze layer is enormous but touched rarely, so archive pricing saves the most; the gold layer is small but read constantly, so any retrieval fee there would swamp the storage saving. Tiering is the second lever you pull after byte reduction, and it only pays off when you respect each tier’s minimum-duration commitment and retrieval latency. This page details the tier assignment and the mechanics of applying it, extending the lifecycle model in the parent guide on cloud cost and storage lifecycle.

#Quick-Reference Table

#How to Assign Tiers Without Getting Burned

A storage tier is a trade of price against latency and retrieval risk. Standard storage costs the most per GB-month but reads instantly with no retrieval fee. Infrequent-access tiers roughly halve the storage rate but add a per-GB retrieval charge and a 30-day minimum. Archive tiers cut the storage rate by an order of magnitude but impose retrieval fees, minimum durations of 90–180 days, and restore times measured in minutes to hours. The discipline is to place each layer where its access recency — not its size — puts it, because retrieval fees are charged on activity while storage savings accrue on volume.

The medallion structure maps cleanly onto these tiers. The bronze layer holds raw source data — untouched Shapefiles, original imagery scenes, sensor dumps — that you keep for reprocessing and provenance but query almost never. It is the largest layer and the best archive candidate, provided reprocessing has settled. The silver layer holds cleaned, compressed GeoParquet that analytical jobs scan on a cadence; its access is periodic and often bursty, which is exactly the pattern Intelligent-Tiering handles without a retrieval penalty. The gold layer holds serving views and tiles read on every dashboard load or map pan, so it stays hot regardless of size.

The choice between Intelligent-Tiering and explicit lifecycle rules turns on predictability. If you can name the schedule — “imagery is hot for its first 30 days, then nobody looks at it” — an explicit rule captures the saving without the per-object monitoring fee that Intelligent-Tiering charges. If access is genuinely unpredictable, as it often is for a silver layer feeding ad-hoc analytics, Intelligent-Tiering earns its fee by moving objects automatically and, critically, never charging retrieval when a cold object is suddenly read again. Format choice interacts with all of this: because GeoParquet and FlatGeobuf let engines read only the fragments a query needs, a partitioned layer can sit in a cooler tier while still serving selective queries cheaply, since each query restores only the partitions it touches.

#Lifecycle Policy Implementation

The function below builds and applies an S3 lifecycle configuration that transitions a prefix through standard, infrequent-access, and archive, with transition ages set safely beyond each tier’s minimum-duration commitment. It is idempotent — re-running it replaces the configuration rather than duplicating rules.

# Requires: boto3>=1.34, python>=3.10
from __future__ import annotations

import boto3
from botocore.exceptions import ClientError


def build_spatial_lifecycle(
    prefix: str,
    *,
    ia_after_days: int = 30,
    archive_after_days: int = 90,
    deep_archive_after_days: int | None = 365,
) -> dict:
    """Construct a lifecycle rule that ages a spatial prefix through tiers.

    Transition ages must clear each tier's minimum-duration commitment to
    avoid early-deletion charges: 30 days for IA, 90 for archive classes.
    """
    if ia_after_days < 30:
        raise ValueError("IA transition must be >= 30 days (minimum duration)")
    if archive_after_days < ia_after_days:
        raise ValueError("archive transition must come after IA transition")

    transitions = [
        {"Days": ia_after_days, "StorageClass": "STANDARD_IA"},
        {"Days": archive_after_days, "StorageClass": "GLACIER"},
    ]
    if deep_archive_after_days is not None:
        if deep_archive_after_days < archive_after_days:
            raise ValueError("deep archive must come after archive transition")
        transitions.append(
            {"Days": deep_archive_after_days, "StorageClass": "DEEP_ARCHIVE"}
        )

    return {
        "ID": f"spatial-tiering-{prefix.strip('/').replace('/', '-')}",
        "Filter": {"Prefix": prefix},
        "Status": "Enabled",
        "Transitions": transitions,
    }


def apply_lifecycle(bucket: str, rules: list[dict]) -> None:
    """Replace the bucket lifecycle configuration with the given rules."""
    s3 = boto3.client("s3")
    try:
        s3.put_bucket_lifecycle_configuration(
            Bucket=bucket,
            LifecycleConfiguration={"Rules": rules},
        )
    except ClientError as exc:
        raise RuntimeError(f"failed to set lifecycle on {bucket}: {exc}") from exc


if __name__ == "__main__":
    rules = [
        # Bronze raw: archive quickly, then deep archive for long retention
        build_spatial_lifecycle("bronze/", ia_after_days=30,
                                archive_after_days=60,
                                deep_archive_after_days=180),
        # Silver analytical: age to IA only; let Intelligent-Tiering handle rest
        build_spatial_lifecycle("silver/", ia_after_days=45,
                                archive_after_days=365,
                                deep_archive_after_days=None),
    ]
    apply_lifecycle("my-spatial-estate", rules)
    print(f"Applied {len(rules)} lifecycle rules")

For the gold serving layer, apply no transition rule at all — leave it in standard so every read is immediate and fee-free. To opt a prefix into Intelligent-Tiering instead of explicit rules, add a transition to the INTELLIGENT_TIERING storage class rather than STANDARD_IA, and omit the archive transitions so the class manages movement itself.

#Validation

After applying a policy, confirm objects actually transition by inspecting per-storage-class metrics rather than trusting the rule was accepted. The CLI check below reports how many bytes now sit in each storage class for a bucket.

aws s3api list-objects-v2 --bucket my-spatial-estate --prefix bronze/ \
  --query "Contents[].StorageClass" --output text | tr '\t' '\n' | sort | uniq -c

Expected result a few days after a 60-day archive rule takes effect on objects older than 60 days: the bronze prefix should report GLACIER (or DEEP_ARCHIVE) for the aged objects and STANDARD only for recent arrivals. If everything still reports STANDARD, the objects have not yet reached the transition age, or the rule prefix does not match the actual key layout — the most common lifecycle mistake.

#Edge Cases and Caveats

Small objects and the IA floor. Infrequent-access classes bill a minimum object size of 128 KB regardless of actual size. A silver layer shattered into thousands of tiny per-tile objects pays the 128 KB minimum on each, so IA can cost more than standard. Consolidate into larger partitioned objects before tiering, or keep small-object layers on standard or Intelligent-Tiering.

Reprocessing against archived bronze. If a schema change or corrected pipeline forces you to reprocess raw data, an archived bronze layer must be restored in full first — paying retrieval on the entire layer plus the wait. Before archiving bronze, confirm the upstream pipeline is stable; if reprocessing is still likely, hold bronze in infrequent-access rather than archive so restores stay instant.

Retrieval latency blocks queries. A query engine reading a partition that has aged into Glacier will fail or block until the object is restored, which for a deep-archive layer can be up to 12 hours. Never place a layer that participates in interactive queries into an archive tier; reserve archive strictly for data whose consumers tolerate a restore delay, and keep a hot manifest or index outside the archive so you can locate objects without restoring the bulk.

#Frequently Asked Questions

Should I use S3 Intelligent-Tiering or write my own lifecycle rules?

Use Intelligent-Tiering when access is unpredictable and objects exceed 128 KB — it moves each object between frequent and infrequent tiers automatically for a small per-object monitoring fee, with no retrieval charge. Write explicit lifecycle rules when you already know the access schedule, such as imagery hot for 30 days then archived, because you avoid the monitoring fee and control transition timing precisely. Many estates use both: Intelligent-Tiering for serving layers, explicit rules for predictable raw archives.

Which medallion layer belongs in which storage tier?

As a default, keep the gold serving layer hot in standard storage, place the silver analytical layer in Intelligent-Tiering or infrequent-access, and send the bronze raw layer to archive once it is no longer being reprocessed. The bronze layer is large but rarely read, so it benefits most from archive pricing; the gold layer is small but read constantly, so retrieval fees there would dominate.

How long does it take to read data back from an archive tier?

It depends on the retrieval option. Glacier Flexible Retrieval offers expedited restores in 1–5 minutes at a premium, standard in 3–5 hours, and bulk in 5–12 hours at the lowest cost. Glacier Deep Archive standard restores take up to 12 hours and bulk up to 48. Because a spatial query cannot complete until the objects are restored, match the tier to the recovery window your users can actually tolerate.

#Related

• Cloud Cost & Storage Lifecycle for Spatial Data — parent guide: cost meters and the cold-tier break-even check
• Object Storage Egress Cost Modelling for Geospatial Data — how transfer cost interacts with tier choice
• Budget Alerting for Geospatial Storage — catching a mis-tiered layer before the bill
• Comparing GeoParquet vs FlatGeobuf Performance — how format read patterns let cooler tiers stay viable
• Understanding Parquet Columnar Storage for GIS — partitioned layouts that restore only the fragments a query needs

← Back to Cloud Cost & Storage Lifecycle for Spatial Data