#Cost-per-GB Tracking for Conversion Pipelines

To track cost per gigabyte for a Shapefile or GeoJSON to GeoParquet pipeline, divide total spend by source gigabytes converted, and decompose that spend into three components: compute, storage, and egress. Compute is vCPU-seconds multiplied by the instance rate; storage is output gigabyte-months at the tier rate; egress is gigabytes moved across the network at the transfer rate. Attribute each component to the individual dataset that caused it, amortise shared batch overhead across jobs, and always divide by source size rather than compressed output so a better codec shows up as a saving, not an apparent cost increase. This page gives the formula, a ready-to-run cost tracker that emits a per-job record, and the edge cases where naive accounting misleads. It expands on the cost-attribution section of Cost & Observability for Conversion Pipelines.

#Quick-Reference Table

#How the Cost Decomposition Works

The instinct to report a single blended dollars-per-gigabyte number is understandable but hides the lever you actually control. A conversion job’s bill is not one thing; it is three costs with different drivers. Compute scales with how much CPU the job burns, which in turn depends on geometry complexity and the compression level chosen — pushing the codec harder, as explored in ZSTD Compression Levels for Geospatial Data, trades more compute now for less storage later. Storage is recurring: you pay for the output GeoParquet every month it sits in the bucket, so a codec that shrinks output bytes pays back repeatedly. Egress is the quiet one — reading a large Shapefile across regions or re-converting the same dataset nightly can dwarf both compute and storage, and it never shows up if you only measure the compressed result.

Attribution is what turns this from an accounting curiosity into an operational tool. Because the resource counters that feed the bill are captured per job — compute seconds, source bytes, bytes read, output bytes — you can rank datasets by cost, spot the handful that consume most of the budget, and decide whether they warrant a cheaper tier, a different schedule, or an on-off toggle. This is the same per-job record that drives the metrics and dashboards described across the parent cost and observability topic; here the record simply carries its cost fields fully populated.

The final subtlety is denominator choice. Reporting cost against source gigabytes keeps the metric honest: the pipeline was asked to convert a 640 MB Shapefile, and that figure does not change when you swap Snappy for ZSTD. If you instead divide by output bytes, improving compression shrinks the denominator and inflates the apparent cost per gigabyte — punishing exactly the optimisation you want to encourage. Track compression savings as a separate, celebrated line: source bytes minus output bytes, in gigabytes and in avoided storage dollars.

#Cost Tracker Implementation

The tracker below computes all three cost components plus amortised overhead, derives dollars per source gigabyte, quantifies compression savings, and returns a flat record ready to append to a ledger or emit as metric dimensions. Rates live in a config object so a pricing change is a single edit.

# python>=3.10
from __future__ import annotations

from dataclasses import dataclass, asdict


@dataclass(frozen=True)
class Rates:
    compute_usd_per_sec: float = 0.0000116   # vCPU-second on the chosen instance
    storage_usd_per_gb_month: float = 0.023  # standard object-storage tier
    egress_usd_per_gb: float = 0.09          # cross-network transfer


@dataclass
class CostRecord:
    dataset_id: str
    source_gb: float
    output_gb: float
    compression_ratio: float
    compute_usd: float
    storage_usd_month: float
    egress_usd: float
    amortised_overhead_usd: float
    total_usd: float
    usd_per_source_gb: float
    gb_saved: float


def build_cost_record(
    *,
    dataset_id: str,
    source_bytes: int,
    output_bytes: int,
    bytes_transferred: int,
    compute_seconds: float,
    batch_fixed_usd: float,
    batch_job_count: int,
    rates: Rates = Rates(),
) -> CostRecord:
    """Compute a per-job cost record for one conversion.

    Overhead is amortised evenly across the batch. `bytes_transferred`
    is the sum of remote bytes read and written for this job.
    """
    if source_bytes <= 0:
        raise ValueError("source_bytes must be positive to attribute cost")
    if batch_job_count <= 0:
        raise ValueError("batch_job_count must be at least 1")

    source_gb = source_bytes / 1024**3
    output_gb = output_bytes / 1024**3
    transferred_gb = bytes_transferred / 1024**3

    compute = compute_seconds * rates.compute_usd_per_sec
    storage = output_gb * rates.storage_usd_per_gb_month
    egress = transferred_gb * rates.egress_usd_per_gb
    overhead = batch_fixed_usd / batch_job_count

    total = compute + storage + egress + overhead
    ratio = round(source_bytes / output_bytes, 3) if output_bytes else 0.0

    return CostRecord(
        dataset_id=dataset_id,
        source_gb=round(source_gb, 4),
        output_gb=round(output_gb, 4),
        compression_ratio=ratio,
        compute_usd=round(compute, 6),
        storage_usd_month=round(storage, 6),
        egress_usd=round(egress, 6),
        amortised_overhead_usd=round(overhead, 6),
        total_usd=round(total, 6),
        usd_per_source_gb=round(total / source_gb, 6),
        gb_saved=round(source_gb - output_gb, 4),
    )


if __name__ == "__main__":
    record = build_cost_record(
        dataset_id="parcels_2026",
        source_bytes=640_000_000,
        output_bytes=210_000_000,
        bytes_transferred=850_000_000,
        compute_seconds=42.0,
        batch_fixed_usd=0.12,
        batch_job_count=30,
    )
    print(asdict(record))

#Validation

Verify the tracker on a known input before trusting its ledger. For the example call above — a 640 MB source compressed to 210 MB, 42 compute seconds, and a batch overhead of $0.12 spread across 30 jobs — the components should land in these ranges:

compute_usd            ~= 0.00049   (42 s x 0.0000116)
storage_usd_month      ~= 0.0045    (0.1956 GB x 0.023)
egress_usd             ~= 0.0712    (0.7916 GB x 0.09)
amortised_overhead_usd  = 0.004     (0.12 / 30)
total_usd              ~= 0.0802
compression_ratio      ~= 3.05      (640 / 210)
usd_per_source_gb      ~= 0.135     (0.0802 / 0.596 GB)
gb_saved               ~= 0.40      (0.596 - 0.196)

The dominant component here is egress, not compute — a common and instructive result. If your usd_per_source_gb is dominated by transfer cost, the highest-leverage fix is usually co-locating conversion with the source bucket rather than tuning the codec. Recompute the record after any rate change and confirm the total moves in the direction the rate did; a total that drifts independently of its inputs means a rate is hardcoded somewhere it should not be.

#Edge Cases and Caveats

Many tiny datasets. When a batch is dominated by small files, fixed overhead — scheduler startup, image pull, idle wait — can exceed the marginal compute and storage cost combined. Even-weighted amortisation then charges a 5 MB Shapefile the same overhead as a 40 GB one, which inflates its cost per gigabyte enormously. Switch to byte-weighted amortisation, or better, batch small inputs together so one worker converts many files and the fixed cost is paid once. The batching mechanics themselves belong to Building Batch Conversion Pipelines with Python.

Retries and partial failures. A job that fails after burning compute, then succeeds on retry, has incurred real cost twice but converted the dataset once. Attribute the wasted compute to the dataset (so retry storms are visible in its cost trend) but count the source gigabytes only once in the denominator. Otherwise a flaky dataset appears artificially cheap per gigabyte precisely because it ran repeatedly.

Cross-region and re-conversion egress. Nightly re-conversion of an unchanged dataset pays the egress cost every run for zero new value. Track transfer bytes per job and flag datasets whose cumulative egress is climbing without a matching change in source size — they are candidates for caching, incremental conversion, or a schedule change rather than a codec tweak.

#Frequently Asked Questions

Should cost per gigabyte be based on source size or output size?

Base the headline metric on source gigabytes, because that is the work the pipeline was asked to do and it stays stable when you change codecs. Report output gigabytes and the compression ratio separately. If you divide cost by output size, a better codec makes cost per gigabyte appear to rise even though you spent less overall, which is exactly backwards for decision-making.

How do I attribute shared batch overhead to individual datasets?

Amortise it. Measure the fixed cost of a batch — scheduler startup, container image pull, idle wait — and divide it across the jobs in that batch, weighted either evenly or by source bytes. Weighting by bytes is fairer when file sizes vary by orders of magnitude, because a 5 MB Shapefile should not absorb the same fixed overhead as a 40 GB one.

How often should unit rates be refreshed?

Review them whenever you change instance family, storage tier, or region, and otherwise on a monthly cadence against the actual cloud bill. Keep rates in versioned configuration so a historical cost record can be recomputed against the rates that were in force when the job ran, rather than being silently rewritten by a later price change.

#Related

• Cost & Observability for Conversion Pipelines — parent topic: signals, dashboards, and SLOs for conversion jobs
• Structured Logging and Metrics for Migration Jobs — the log and metric plumbing that carries these cost records
• Building Batch Conversion Pipelines with Python — where batching decisions change how overhead amortises
• ZSTD Compression Levels for Geospatial Data — the compute-versus-storage trade-off behind compression savings

← Back to Cost & Observability for Conversion Pipelines