Data Compression
  1. Home
  2. Geospatial Storage Fundamentals & Format Comparison
  3. Cloud-Optimized GeoTIFF for Raster Workloads
  4. Validating COG Structure in CI
Section: Geospatial Storage Fundamentals & Format Comparison 14 min read

Validating COG Structure in CI

A Cloud-Optimized GeoTIFF is an ordinary GeoTIFF with three structural constraints — internal tiling, internal overviews, and front-loaded headers — and nothing in the file’s name, extension, or metadata tells you whether it has them. Every GeoTIFF reader opens a broken COG without complaint, renders it correctly, and quietly costs you a hundred times the bytes on every remote read. That is why this belongs in a pipeline gate. This page extends Cloud-Optimized GeoTIFF for raster workloads.

Quick Reference

Check Requirement Failure symptom Primary Use Case
Internal tiling Tiled, typically 512×512 Window read pulls full-width strips Any remotely-read raster
Internal overviews Present, powers of two, inside the file Zoomed-out reads decode full resolution Tile services and previews
Header placement IFDs and tile offsets at the front Many round trips before the first pixel High-latency object storage
Compression ZSTD or DEFLATE, per tile Whole-file compression breaks tile seeking Cost and transfer efficiency
Range read Bytes ∝ window, not file size The empirical proof the above worked Final gate before publish

The Three Constraints and Why Each Exists

Tiling instead of strips. A classic GeoTIFF stores rows of pixels in strips spanning the full image width. Reading a 512×512 window from a 40,000-pixel-wide image therefore means reading 512 strips of 40,000 pixels — about 80 times more data than the window contains. Internal tiling stores the image as a grid of small rectangles, so a window read touches only the tiles it overlaps.

Internal overviews. A viewer showing the whole scene at 1,000 pixels wide does not need 40,000-pixel data. Overviews are pre-computed reduced-resolution copies stored inside the same file; without them the reader must fetch and downsample full resolution for every zoomed-out view. Sidecar overview files (.ovr) work locally and are useless in object storage, because the reader would have to discover and fetch a second object.

Header placement. A TIFF’s image file directories and tile offset tables can legally live anywhere in the file, and many writers put them at the end. A remote reader then fetches the first bytes, finds a pointer to the end, fetches the end, and only then knows where the tiles are — several round trips before a single pixel. A COG places all of it at the front so one modest range read reveals the entire layout.

What a window read costs on a striped file versus a cloud-optimized one The upper layout is a striped GeoTIFF with its headers at the end: a reader fetches the file start, follows a pointer to the trailing directory, fetches that, and then must read full-width strips covering the window's rows, transferring far more than the window contains. The lower layout is a cloud-optimized GeoTIFF: headers and tile offsets sit at the front so one small read reveals the layout, overviews follow, and the full-resolution tiles are stored as a grid so the reader fetches only the four tiles its window overlaps. Striped, headers at the end — three round trips, 80× over-fetch full-width strips, in row order headers read 1: file start → pointer · read 2: trailing directory · read 3: 512 full-width strips for a 512-pixel window Cloud-optimized — one header read, then four tiles headers + tile offsets overviews full-resolution tiles, 512 × 512, in a grid read 1: the shaded header block · read 2: the four shaded tiles the window overlaps — nothing else moves Both files contain identical pixels and both open correctly in every desktop tool. The difference is invisible until it appears on the egress bill, which is why it belongs in CI.

The Gate

python
# Requires: rasterio>=1.3, rio-cogeo>=5.3  (Python 3.10+)
from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path

import rasterio
from rio_cogeo.cogeo import cog_validate


@dataclass(frozen=True)
class CogReport:
    path: str
    tiled: bool
    block: tuple[int, int] | None
    overview_levels: tuple[int, ...]
    compression: str | None
    errors: tuple[str, ...]
    warnings: tuple[str, ...]

    @property
    def ok(self) -> bool:
        return not self.errors


MIN_OVERVIEW_LEVELS = 3
ACCEPTED_COMPRESSION = {"ZSTD", "DEFLATE", "LZW"}


def validate(path: Path, *, require_overviews: bool = True) -> CogReport:
    """Structural validation strict enough to be a gate.

    cog_validate covers the specification; the extra assertions here cover the
    things that are legal but still make remote reads expensive.
    """
    if not path.exists():
        raise FileNotFoundError(path)

    valid, errors, warnings = cog_validate(str(path), quiet=True)
    errors = list(errors)

    with rasterio.open(path) as src:
        tiled = bool(src.profile.get("tiled"))
        block = (src.profile.get("blockxsize"), src.profile.get("blockysize")) if tiled else None
        overviews = tuple(src.overviews(1))
        compression = src.profile.get("compress")

        if not tiled:
            errors.append("not internally tiled — window reads will pull full-width strips")
        elif block and (block[0] < 256 or block[1] < 256):
            errors.append(f"block size {block} is too small — per-tile overhead dominates")

        if require_overviews and len(overviews) < MIN_OVERVIEW_LEVELS:
            errors.append(
                f"only {len(overviews)} overview level(s); zoomed-out reads will "
                f"decode full resolution"
            )
        if path.with_suffix(path.suffix + ".ovr").exists():
            errors.append("overviews are in a sidecar .ovr — remote readers will not find them")

        if compression and compression.upper() not in ACCEPTED_COMPRESSION:
            errors.append(f"compression {compression} is not a per-tile codec")

    return CogReport(
        path=str(path), tiled=tiled, block=block, overview_levels=overviews,
        compression=compression, errors=tuple(errors), warnings=tuple(warnings),
    )


def gate(paths: list[Path]) -> int:
    """Return a process exit code: 0 when every file passes."""
    failed = 0
    for path in paths:
        report = validate(path)
        if report.ok:
            print(f"PASS {path.name} — tiled {report.block}, "
                  f"{len(report.overview_levels)} overviews, {report.compression}")
        else:
            failed += 1
            print(f"FAIL {path.name}")
            for error in report.errors:
                print(f"     {error}")
    return 1 if failed else 0

Validation

Structural checks prove the file is shaped correctly; a range read proves it behaves correctly. Run both, because a file can satisfy every structural assertion and still be served from a bucket that ignores range requests.

bash
# Structural — the specification check plus the extra assertions
rio cogeo validate scene.tif
# scene.tif is a valid cloud optimized GeoTIFF

# Empirical — bytes transferred for a small window over HTTP
CPL_DEBUG=ON CPL_CURL_VERBOSE=YES \
  gdal_translate -srcwin 20000 15000 512 512 \
  /vsicurl/https://cdn.example.com/scene.tif /tmp/window.tif 2>&1 \
  | grep -Eo 'Downloading [0-9]+' | awk '{s+=$2} END {print s" bytes"}'
# Expect: low hundreds of kilobytes for a 512×512 window.
# A figure in the tens of megabytes means the structure is wrong, or the
# bucket is not honouring Range and the whole file came down.

Healthy ranges: a 512×512 window from a multi-gigabyte scene should transfer under about 1 MB, with two to four HTTP requests. Overview reads for a full-scene preview should transfer under about 500 KB.

Where the structural gate sits in a raster pipeline A pipeline flows from source imagery through a conversion step that writes a cloud-optimized GeoTIFF, into a validation gate. Files that pass the gate are uploaded to object storage and referenced by consumers. Files that fail are returned to the conversion step with the specific structural defect named, and never reach storage. A note observes that after upload the same defect costs a rewrite plus a cache invalidation, which is why the gate sits before rather than after. The gate sits before publish, where a defect costs one rebuild Source imagery scenes, mosaics, DEMs Convert to COG tile · overviews · compress Structural gate + one real range read Publish to storage consumers reference it pass fail — with the defect named Before publish, a defect costs one rebuild. After publish, it costs a rewrite plus a cache invalidation plus every consumer that has already cached the object key. Two seconds of validation per file against an incident that surfaces weeks later as unexplained egress. Structure is necessary and not sufficient A two-by-two matrix of structural validity against measured range-read behaviour. A file that is structurally valid and reads efficiently is the goal. A structurally invalid file always reads badly. The interesting cell is structurally valid but reading badly, which happens when the bucket ignores range requests, a content delivery network fetches whole objects, or the reader is configured without range support. Only an end-to-end measurement distinguishes it. Why the gate includes a real range read reads efficiently reads badly structurally valid the goal the interesting cell bucket, CDN or reader config structurally invalid cannot happen caught by the static checks Static assertions alone would pass the top-right cell, which is where most production incidents live.

Edge Cases and Caveats

Files that pass validation and still read badly. Structure is necessary, not sufficient: a bucket that does not honour Range, a CDN that fetches whole objects, or a reader configured without /vsicurl caching will all defeat a perfectly-formed COG. That is why the gate includes an actual range read rather than only static assertions — the same reasoning behind verifying range request configuration for tile archives.

Overviews built with the wrong resampling. Nearest-neighbour overviews of continuous data produce visibly wrong previews — aliased coastlines, speckled elevation — while passing every structural check. Choose average or cubic for continuous rasters and nearest only for categorical ones, and record the choice in the file’s metadata.

Block size chosen without regard to the reader. 256×256 blocks make a tile service efficient and an analytical full-scan slow; 1024×1024 does the reverse. Match the block size to the dominant read, and note that this is the raster form of the same granularity trade as row group sizing.

Frequently Asked Questions

What actually makes a GeoTIFF cloud-optimized?

Three structural properties, none of which is visible from the file extension. The image data must be organised in tiles rather than strips, so a window read touches a small rectangle instead of full-width scanlines. Reduced-resolution overviews must be stored inside the file, so a zoomed-out read does not decode full resolution. And the headers — image directories and tile offset tables — must sit at the front, so a reader learns the whole layout from one small range request before fetching any pixels.

Can a file be a valid GeoTIFF but a broken COG?

Constantly, and that is the reason to validate. Every COG is a valid GeoTIFF, so every GeoTIFF reader opens it happily and every desktop tool renders it correctly. The failure is purely economic: a striped file with headers at the end forces a remote reader to make many round trips and download far more than it needs, so the symptom is a slow tile service or a large egress bill rather than an error message.

Why validate in CI rather than at read time?

Because at read time the file has already been written, uploaded, and referenced by consumers, so fixing it means a rewrite and a cache invalidation. Structure is decided entirely by the writer, so the writer’s pipeline is where a defect is cheap to catch. A validation step that takes two seconds per file and fails the build turns a class of production incident into a red pipeline.

← Back to Cloud-Optimized GeoTIFF for Raster Workloads