Data Compression
  1. Home
  2. Compression, Chunking & Spatial Indexing
  3. Zarr and Chunked Array Storage for Raster
  4. Blosc Codec Selection for Zarr Geospatial Arrays
Section: Compression, Chunking & Spatial Indexing: The Foundation of Cloud-Native Geospatial Storage 14 min read

Blosc Codec Selection for Zarr Geospatial Arrays

Three choices in a Zarr codec pipeline — shuffle filter, compressor, level — and their importance runs in exactly that order. The shuffle filter is nearly free and frequently worth more than several compression levels; the compressor decides whether you are optimising for bytes or for decode speed; the level, on geospatial raster, is the one that matters least and gets tuned first. This page continues Zarr and chunked array storage for raster.

Quick Reference

Data Shuffle Compressor Level Primary Use Case
float32 continuous (temperature, NDVI) SHUFFLE ZSTD 3–5 Cloud-stored analytical cubes
int16 scaled values SHUFFLE ZSTD 3 Satellite reflectance archives
uint8 classification BITSHUFFLE ZSTD 5 Land-cover and mask layers
Boolean masks BITSHUFFLE ZSTD 5 Quality flags, valid-data masks
Anything local and latency-critical SHUFFLE LZ4 5 Page-cached or NVMe-backed reads
Already-compressed payloads NOSHUFFLE none — Pre-encoded imagery tiles

What Each Stage Actually Does

Shuffle is a byte transposition, not a compression. Given n values of k bytes each, it emits all the first bytes, then all the second bytes, and so on. Nothing is removed; the data is merely reordered so that bytes of equal significance become neighbours.

The effect on a raster is large because of how floating-point numbers are laid out. In a chunk of temperatures spanning 280–300 K, every value has the same exponent and a nearly identical high mantissa byte, while the low byte varies with sensor noise. Interleaved, the byte stream reads [stable, stable, varying, noise, stable, stable, varying, noise, …] — a pattern with period four that most compressors model poorly. Shuffled, it reads as one long run of nearly-identical bytes, then another, then the noisy ones at the end. The first two runs compress to almost nothing.

Bit shuffle does the same at bit granularity, which pays when values are small integers or booleans and the interesting structure is below the byte level. It costs more CPU than byte shuffle and is wasted on wide floats.

The compressor then does ordinary entropy coding on the reordered stream. LZ4 is extremely fast and moderately effective; ZSTD is slower and noticeably smaller. Which one wins depends entirely on whether the chunk is arriving over a network.

What the shuffle filter does to a run of float32 values The upper strip shows four float32 values stored in memory order: for each value, a stable high byte, a second stable byte, a varying byte, and a noisy low byte, repeating with period four. The lower strip shows the same bytes after shuffling: all four high bytes together, then all four second bytes, then all the varying bytes, then all the noisy bytes. The first two groups are now long runs of near-identical values that a compressor collapses, while the noisy group remains incompressible either way. Memory order — four float32 values, period-4 interleave stable · stable · varying · noise, repeating — the compressor sees a pattern it models poorly After byte shuffle — grouped by significance all high bytes all second bytes all varying bytes all noisy bytes two long near-constant runs the entropy coder collapses, plus a noisy tail that was never compressible Nothing was removed and nothing was approximated — the bytes were only reordered. On typical float32 geophysical raster this alone is worth 10–40% before the compressor runs.

Choosing and Measuring

python
# Requires: numcodecs>=0.13, numpy>=1.26, zarr>=3.0  (Python 3.10+)
from __future__ import annotations

import time
from dataclasses import dataclass

import numpy as np
from numcodecs import Blosc


@dataclass(frozen=True)
class CodecResult:
    label: str
    ratio: float
    encode_mb_s: float
    decode_mb_s: float

    def score(self, link_mb_s: float) -> float:
        """Effective throughput over a link of the given speed.

        A chunk must both arrive and decode; whichever is slower governs, so a
        codec that halves the bytes is only a win if its decode still outruns
        the link at the new size.
        """
        arrival = link_mb_s * self.ratio          # effective MB/s of source data
        return min(arrival, self.decode_mb_s)


def shuffle_for(dtype: np.dtype) -> int:
    """Byte shuffle for wide numerics, bit shuffle for narrow ones."""
    if dtype.kind == "b" or dtype.itemsize == 1:
        return Blosc.BITSHUFFLE
    return Blosc.SHUFFLE


def evaluate(sample: np.ndarray, candidates: dict[str, Blosc]) -> list[CodecResult]:
    """Measure ratio and both throughputs on a real chunk, not synthetic data."""
    if sample.nbytes < (1 << 20):
        raise ValueError("use a realistic chunk-sized sample — small samples mislead")
    raw = sample.tobytes()
    megabytes = len(raw) / 1e6

    results = []
    for label, codec in candidates.items():
        start = time.perf_counter()
        encoded = codec.encode(sample)
        encode_s = time.perf_counter() - start

        start = time.perf_counter()
        decoded = codec.decode(encoded)
        decode_s = time.perf_counter() - start

        if not np.array_equal(np.frombuffer(decoded, dtype=sample.dtype).reshape(sample.shape), sample):
            raise RuntimeError(f"{label} did not round-trip — refusing to recommend it")

        results.append(CodecResult(
            label=label,
            ratio=len(raw) / len(encoded),
            encode_mb_s=megabytes / max(encode_s, 1e-9),
            decode_mb_s=megabytes / max(decode_s, 1e-9),
        ))
    return sorted(results, key=lambda r: -r.ratio)


def recommend(sample: np.ndarray, link_mb_s: float) -> CodecResult:
    """Pick the codec with the best effective throughput over this link."""
    sh = shuffle_for(sample.dtype)
    candidates = {
        "lz4 l5":  Blosc(cname="lz4", clevel=5, shuffle=sh),
        "zstd l1": Blosc(cname="zstd", clevel=1, shuffle=sh),
        "zstd l3": Blosc(cname="zstd", clevel=3, shuffle=sh),
        "zstd l5": Blosc(cname="zstd", clevel=5, shuffle=sh),
        "zstd l9": Blosc(cname="zstd", clevel=9, shuffle=sh),
        "zstd l3 noshuffle": Blosc(cname="zstd", clevel=3, shuffle=Blosc.NOSHUFFLE),
    }
    return max(evaluate(sample, candidates), key=lambda r: r.score(link_mb_s))

Validation

Report ratio and decode throughput in the same table, always. A codec chosen on ratio alone will eventually be chosen at level 19, where compression is slow, decode is mediocre, and the extra four per cent buys nothing.

python
# Requires: the module above — measure on a real chunk from the real array
import zarr
import numpy as np

store = zarr.open("temperature.zarr", mode="r")
sample = np.asarray(store["t2m"][0:24, 0:512, 0:512])   # one chunk-sized block

for result in evaluate(sample, {...}):
    print(f"{result.label:<20} ratio {result.ratio:5.2f}x  "
          f"encode {result.encode_mb_s:7.0f} MB/s  decode {result.decode_mb_s:7.0f} MB/s")

Expected ranges on float32 geophysical raster with byte shuffle: LZ4 around 2.2× at 2,500 MB/s decode; ZSTD level 3 around 3.1× at 700 MB/s; ZSTD level 9 around 3.4× at 620 MB/s but four times the encode cost. Without shuffle, every ratio drops by roughly a third — which is the number that settles the argument about whether shuffle is worth its CPU.

Ratio against decode throughput, with the link speed that decides between them Six codec settings plotted with compression ratio on one axis and decode throughput on the other. LZ4 sits at low ratio and very high decode speed. ZSTD levels one through nine climb in ratio while falling in decode speed, with sharply diminishing ratio returns above level three. ZSTD level three without the shuffle filter sits well below the shuffled version at the same speed, showing that the filter is worth more than several levels. A vertical band marks a typical cloud link speed, to the left of which bytes on the wire dominate and to the right of which decode does. Ratio against decode speed — float32 geophysical chunk network-bound smaller wins decode-bound faster wins lz4 l5 2.2× zstd l1 2.7× zstd l3 3.1× zstd l5 zstd l9 zstd l3, no shuffle 2.1× decode throughput → ratio → The dashed line is the shuffle filter alone: same compressor, same level, a third more ratio for almost no CPU. Budget worker memory on the uncompressed chunk, not the stored one A four-megabyte stored chunk that compresses eight to one expands to thirty-two megabytes when decoded. With sixteen concurrent workers each holding one decoded chunk plus an output buffer, the real requirement is over five hundred megabytes, not sixty-four. Sizing a worker from the stored figure is the most common cause of an out-of-memory failure in an otherwise correct Zarr pipeline. What sixteen workers actually need Stored chunk 4 MB × 16 = 64 MB Decoded chunk 32 MB × 16 = 512 MB Plus output buffers ~640 MB is the real requirement A high compression ratio is a memory liability at read time, not only a storage win. Measure the uncompressed chunk before choosing the concurrency.

Edge Cases and Caveats

Benchmarking on synthetic data. Random arrays are incompressible and constant arrays compress infinitely; neither tells you anything. Always sample a real chunk from the real array, at the real chunk size, because compressibility is a property of the data and varies enormously between a temperature field and a classification mask.

Ignoring the memory cost of a high ratio. A chunk that compresses 8× expands to eight times its stored size in a worker’s memory, and concurrency multiplies that. Budget worker memory on the uncompressed chunk size times the concurrency — the constraint that Zarr chunk sizing also has to respect.

Compressing data whose precision was never reduced. If the low mantissa bits are sensor noise, no codec removes them and every level costs CPU for nothing. Reducing precision to the data’s real accuracy first — the raster analogue of coordinate precision reduction — often doubles the achievable ratio at any level.

Frequently Asked Questions

What does the shuffle filter do and why does it help so much?

It transposes the bytes of an array so that the first byte of every value sits together, then the second byte of every value, and so on. In a float32 raster the first byte carries the sign and high exponent bits, which barely change between neighbouring pixels, while the last byte is noise. Interleaved, a compressor sees an alternating pattern it cannot model; shuffled, it sees a long run of near-identical bytes followed by a noisy run, and the first run compresses very well. The cost is a fast in-memory permutation.

Should I use LZ4 or ZSTD inside Blosc?

Decide by where the bottleneck is. LZ4 decompresses at several gigabytes per second, so it is effectively free and the right choice when chunks arrive over a fast local link or come from a page cache. ZSTD produces noticeably smaller chunks at maybe a quarter of LZ4’s decode speed, which wins whenever bytes cross a network — the smaller payload arrives sooner than the faster decode could have finished. For cloud object storage, ZSTD at a low level is usually correct.

Does a higher compression level help on raster data?

Much less than people expect, because a raster chunk’s compressibility is dominated by the shuffle filter and by how noisy the low-order bits are, not by how hard the entropy coder works. On typical float32 geophysical data, moving from ZSTD level 1 to level 5 buys perhaps eight per cent and costs twice the compression time; moving to level 9 buys another two per cent for four times the time again. Spend the effort on the shuffle and on the data’s precision instead.

← Back to Zarr and Chunked Array Storage for Raster