Data Compression
  1. Home
  2. Compression, Chunking & Spatial Indexing
  3. Zarr and Chunked Array Storage for Raster
  4. Zarr v3 vs COG for Multidimensional Data
Section: Compression, Chunking & Spatial Indexing: The Foundation of Cloud-Native Geospatial Storage 13 min read

Zarr v3 vs COG for Multidimensional Data

Three questions decide it: how many axes do readers slice, must the ecosystem open it directly, and does the dataset grow by appending? Everything else — file size, compression, tooling preference — is secondary and usually cancels out. This page turns the choice into a short set of tests rather than a matter of taste, extending Zarr and chunked array storage for raster and Cloud-Optimized GeoTIFF for raster workloads.

Quick Reference

Test Answer → COG Answer → Zarr Primary Use Case
Axes readers slice Two (y, x) Three or more (t, band, z…) Scene delivery vs cube analytics
Direct ecosystem access Required Not required Desktop GIS and tile servers
Growth pattern Reprocessed in whole scenes Appended along an axis Rolling archives
Per-object granularity One file per scene One object per chunk Object count management
Overviews / pyramids Built in Separate arrays or absent Zoomed-out browsing

The Three Tests, in Order

Test one: how many axes get sliced. This is the decisive one. A COG is a two-dimensional structure with a band list attached; it can chunk in y and x and nothing else. Ask for one spectral band across a whole scene and it reads interleaved data; ask for one pixel across ten thousand dates and it opens ten thousand files. Zarr chunks along every axis, so a time-series read touches a handful of objects. If a third axis is genuinely part of the query, that settles it.

Test two: who opens the file. COG’s decisive advantage is that it is a GeoTIFF. Every desktop GIS, every tile server, every raster library, and every third party you might publish to opens it without any new dependency. A Zarr store needs a reader that speaks Zarr, which the scientific Python stack does and much of the wider geospatial ecosystem still does not. If the deliverable must be openable by an arbitrary consumer, this test overrides test one.

Test three: how the data grows. A rolling archive that gains a time step each day appends one chunk row to a Zarr array — a metadata update and some new objects. The COG equivalent is a new file per date, which is fine, or an extension of an existing file, which is a rewrite. Where appending along an axis is the natural operation, Zarr models it and COG does not.

The three tests, applied in order A decision tree. The first question asks whether readers slice more than two axes; a no leads directly to Cloud-Optimized GeoTIFF. A yes leads to the second question, whether arbitrary consumers must open the files directly. A yes there leads to shipping both formats — a Zarr cube for analysis and per-date COGs for the ecosystem. A no leads to the third question, whether the dataset grows by appending; either answer leads to Zarr, but appending makes the case decisive rather than merely preferable. 1 · Do readers slice more than two axes? time, band, depth, ensemble member no Cloud-Optimized GeoTIFF and stop — the ecosystem is the win yes 2 · Must arbitrary consumers open it directly? desktop GIS · tile servers · external partners no 3 · Does the dataset grow by appending? a new time step each day, indefinitely Zarr — decisive if yes, preferable if no appending is a metadata update, not a rewrite yes Ship both Zarr cube for analysis, per-date COGs for the ecosystem

Generating Both from One Source

Where test two forces the issue, the honest answer is usually to publish both rather than to compromise. The cost is storage and one extra pipeline stage.

python
# Requires: xarray>=2024.7, rioxarray>=0.17, zarr>=3.0, rio-cogeo>=5.3  (Python 3.10+)
from __future__ import annotations

from pathlib import Path

import xarray as xr
from rio_cogeo.cogeo import cog_translate
from rio_cogeo.profiles import cog_profiles


def publish_cube(dataset: xr.Dataset, variable: str, zarr_target: Path) -> None:
    """The analytical copy: chunked along every axis readers slice."""
    if variable not in dataset:
        raise KeyError(f"{variable!r} not in the dataset")
    array = dataset[variable]
    if "time" not in array.dims:
        raise ValueError("a cube without a time axis does not need Zarr")

    dataset.to_zarr(
        zarr_target, mode="w", consolidated=True,
        encoding={variable: {"chunks": (365, 128, 128)}},
    )


def publish_scenes(dataset: xr.Dataset, variable: str, cog_dir: Path) -> list[Path]:
    """The ecosystem copy: one COG per time step, openable by anything."""
    cog_dir.mkdir(parents=True, exist_ok=True)
    profile = cog_profiles.get("deflate")
    profile.update(blockxsize=512, blockysize=512)

    written: list[Path] = []
    for stamp in dataset.time.values:
        slice_ = dataset[variable].sel(time=stamp)
        if slice_.rio.crs is None:
            raise ValueError("each scene must carry a CRS before export")

        date = str(stamp)[:10]
        staging = cog_dir / f".{date}.tmp.tif"
        target = cog_dir / f"{variable}_{date}.tif"
        try:
            slice_.rio.to_raster(staging, driver="GTiff")
            cog_translate(
                staging, target, profile,
                overview_level=5, overview_resampling="average",
                quiet=True, in_memory=False,
            )
        finally:
            staging.unlink(missing_ok=True)
        written.append(target)
    return written

Validation

Prove the choice with the query that motivated it, not with a size comparison. The two formats are within a few per cent on size for the same data and codec; they differ by orders of magnitude on the read that matters.

python
# Requires: xarray>=2024.7, rioxarray>=0.17 — time the query that drove the decision
import time
import xarray as xr
import rioxarray  # noqa: F401  (registers the .rio accessor)
import glob

start = time.perf_counter()
cube = xr.open_zarr("temperature.zarr", consolidated=True)
series_zarr = cube["t2m"].sel(lat=51.45, lon=-2.58, method="nearest").load()
zarr_seconds = time.perf_counter() - start

start = time.perf_counter()
values = []
for path in sorted(glob.glob("cogs/t2m_*.tif")):        # one open per date
    with xr.open_dataarray(path, engine="rasterio") as scene:
        values.append(float(scene.sel(x=-2.58, y=51.45, method="nearest").values))
cog_seconds = time.perf_counter() - start

print(f"20-year point series — Zarr {zarr_seconds:.2f}s · COGs {cog_seconds:.1f}s")

Expected ranges on a 7,300-date archive: Zarr in single-digit seconds, per-date COGs in the tens of minutes, and a single-date continental read where the ordering reverses — COG in well under a second, Zarr a few seconds depending on chunk shape.

Where each format puts the boundaries, and what that costs each query On the left, a per-date COG archive: one file per date, each internally tiled, so a single-date read opens one file and reads a few tiles while a point time series must open every file in the archive. On the right, a Zarr store: one logical array cut into chunks along time and space, so a point time series touches a short column of chunks while a single-date continental read touches a wide slab of them. Neither structure is better; they place the cheap boundary in different places. COG archive — the file boundary is time day 1 day 2 day 3 …7,300 one date: open 1 file, read a few tiles one point, all dates: open 7,300 files every reader in the ecosystem can open any one of them Zarr store — the chunk boundary is chosen one point, all dates: a short column of chunks one date, whole map: a wide slab of chunks Neither layout is better. Each makes one class of query cheap by making another expensive. COG fixes the boundary at the file; Zarr lets you place it, which is the whole advantage and the whole risk. A Zarr store chunked one-date-per-chunk has all of COG's read profile and none of its ecosystem. What sharding fixes about fine chunk grids The same four-gigabyte array cut into one hundred and twenty-eight kilobyte chunks. Without sharding, each chunk is its own object, so the store is thirty-two thousand objects and every listing, lifecycle rule and per-request charge scales with that number. With sharding, many chunks share one object and the reader range-reads within it, so the same chunk grid becomes five hundred objects while the read granularity is unchanged. Same chunk grid, two object counts Without sharding 32,000 objects one object per chunk — listing and per-request cost track the chunk count With sharding 500 objects many chunks per object; the reader range-reads inside a shard Read granularity is identical in both rows — only the object count changed.

Edge Cases and Caveats

Object count on fine chunk grids. A 4 GB Zarr store cut into 128 KB chunks is 32,000 objects, and listing, lifecycle rules, and per-request charges all scale with that. Zarr v3’s sharding lets many chunks share one object, which is the direct answer; without it, keep chunks in the megabyte band as chunk shape selection recommends, and watch the request cost model.

Overviews are not free in Zarr. A COG carries its pyramid internally and every reader uses it automatically. A Zarr store has no built-in notion of overviews: you build a multiscale group of separate arrays and consumers must know to use it. If zoomed-out browsing matters, that is real work, and it is work COG has already done.

A Zarr store chunked like a COG. Writing one chunk per date reproduces exactly the COG read profile while giving up the ecosystem — the worst of both. If the chunk grid ends up one-date-deep, that is the signal that test one was answered wrongly and COG was the right format all along.

Frequently Asked Questions

Can a COG store more than two dimensions?

It can hold multiple bands, and that covers a spectral axis of modest size. What it cannot do is chunk along that axis, so a query for one band across all pixels still reads interleaved data, and a query along a time axis means opening one file per time step. Bands are a list, not a dimension you can slice efficiently, which is exactly the distinction that decides between the two formats.

Is Zarr v3 different enough from v2 to matter for this decision?

For the format choice, no — both are chunked N-dimensional arrays in object storage. Version 3 tightens the specification, makes the codec pipeline explicit and extensible, standardises consolidated metadata, and adds sharding so many small chunks can share one object. Sharding is the practically important one, because it removes the main operational objection to fine chunk grids, which was the object count.

Should I convert an existing COG archive to Zarr?

Only if the queries you cannot currently answer are the reason. A COG archive that serves spatial windows well is not improved by conversion, and it loses the ecosystem support that made it useful. The case for conversion is specific: readers want time series at points, the archive has thousands of dates, and every such query currently opens thousands of files. Convert for that, and keep the COGs for everything else.

← Back to Zarr and Chunked Array Storage for Raster