#Newline-Delimited GeoJSON for Streaming Pipelines

Put one complete GeoJSON feature on each line, drop the wrapping FeatureCollection, and a file that previously had to be parsed whole becomes a stream you can process with bounded memory, split across workers, and restart from an offset. The change is small enough to apply at a pipeline boundary in a few lines of code, and it is the difference between a 40 GB export that needs a 90 GB machine and one that needs 200 MB. This page belongs to GeoJSON overhead and serialization costs, which covers why the format is expensive; this one covers how to stop it being unworkable.

#Quick Reference

#Why JSON Resists Streaming

JSON is a self-describing grammar with no framing layer. Inside a FeatureCollection, the only thing marking the end of a feature is a } at the right nesting depth — and to know the depth you must have tracked every brace, bracket, and quoted string since byte zero, including escapes inside strings that look like structure. A parser cannot skip ahead, cannot start in the middle, and cannot know how much memory it needs until it has finished.

That is why json.load on a 40 GB export allocates tens of gigabytes: the library is doing the only correct thing, which is to build the whole value before returning it. Streaming JSON parsers avoid the allocation but hand the caller a state machine — events for “started an object”, “saw a key”, “started an array” — and reconstructing features from that is exactly the work the format should have done.

A newline solves it because a newline cannot occur inside a compact JSON scalar or structure: string content escapes it as \n, and whitespace between tokens is removed. So a line break is an unambiguous record boundary that requires no parsing to find.

The practical consequences follow immediately. Splittability: any worker can be handed a byte range, skip to the first newline, and process from there, so conversion parallelises without coordination. Restartability: a job that dies at 31 GB records its offset and resumes, instead of starting over. Appendability: a capture feed appends a line rather than rewriting a document — the property that makes it a natural transport for change data capture.

#Converting and Consuming

# Requires: geopandas>=1.0, pyarrow>=16, shapely>=2.0  (Python 3.10+)
from __future__ import annotations

import json
from pathlib import Path
from typing import Iterator

import geopandas as gpd
import pyarrow as pa
import pyarrow.parquet as pq
from shapely.geometry import shape


def to_ndjson(collection: Path, target: Path, *, chunk: int = 1 << 20) -> int:
    """Split a FeatureCollection into one compact JSON object per line.

    Done at the pipeline boundary, this is the last time the document has to be
    handled as a whole — and on inputs that already exceed memory, prefer a
    streaming JSON parser here rather than json.load.
    """
    payload = json.loads(collection.read_text(encoding="utf-8"))
    features = payload.get("features")
    if not isinstance(features, list):
        raise ValueError(f"{collection} is not a FeatureCollection")

    written = 0
    with open(target, "w", encoding="utf-8") as handle:
        for feature in features:
            # separators removes the whitespace that would otherwise let a
            # pretty-printer smuggle a newline into the middle of a record.
            handle.write(json.dumps(feature, separators=(",", ":"), ensure_ascii=False))
            handle.write("\n")
            written += 1
    return written


def stream_features(path: Path, *, start_offset: int = 0) -> Iterator[tuple[int, dict]]:
    """Yield (offset_after_line, feature) so a caller can checkpoint precisely."""
    with open(path, "r", encoding="utf-8") as handle:
        if start_offset:
            handle.seek(start_offset)
        for line_no, line in enumerate(handle, start=1):
            line = line.strip()
            if not line:
                continue
            try:
                yield handle.tell(), json.loads(line)
            except json.JSONDecodeError as exc:
                raise ValueError(
                    f"{path}: line {line_no} is not valid JSON ({exc.msg}); a "
                    f"pretty-printed feature was probably split across lines"
                ) from exc


def ndjson_to_geoparquet(
    source: Path, target: Path, *, batch: int = 100_000, crs: str = "EPSG:4326"
) -> int:
    """Convert a line-delimited stream into GeoParquet in bounded memory."""
    writer: pq.ParquetWriter | None = None
    rows: list[dict] = []
    total = 0
    try:
        for _, feature in stream_features(source):
            rows.append({
                **(feature.get("properties") or {}),
                "geometry": shape(feature["geometry"]),
            })
            if len(rows) >= batch:
                total += _flush(rows, target, crs, writer)
                writer = writer or pq.ParquetWriter(target, _schema(rows, crs))
                rows = []
        if rows:
            total += _flush(rows, target, crs, writer)
    finally:
        if writer is not None:
            writer.close()
    return total

#Validation

Two properties are worth asserting mechanically, because both fail silently: every line must be a complete JSON object, and the feature count must survive the conversion.

# Line count equals feature count — a pretty-printed input inflates this wildly
wc -l parcels.geojsonl
# 12,041,882

# Every line parses independently; report the first that does not
awk 'NR%1==0 { print }' parcels.geojsonl \
  | python3 -c 'import sys,json
for i, line in enumerate(sys.stdin, 1):
    try: json.loads(line)
    except Exception as e: print(f"line {i}: {e}"); break
else: print("all lines parse")'

Healthy ranges: line count exactly equals the source feature count; the largest single line should sit well under 10 MB (a larger one usually means a multipolygon that should have been simplified); and peak memory during conversion should be roughly the batch size times the average feature size, independent of the input’s total size.

#Edge Cases and Caveats

Pretty-printed input. A source that emits indented JSON puts literal newlines inside each feature, so a line-based reader sees fragments and every line fails to parse. The symptom is unmistakable once you look, but a naive pipeline reports it as corrupt data. Always serialise with compact separators, and validate line count against feature count before proceeding.

Encoding that is not UTF-8. Legacy exports frequently arrive in a regional code page, and a byte that is not valid UTF-8 will fail somewhere in the middle of a multi-hour job. Decode explicitly with a known encoding at the boundary rather than relying on the platform default — the same class of problem as DBF encoding and field name truncation.

Treating it as a destination. Line-delimited GeoJSON fixes framing and nothing else. Coordinates are still decimal text several times larger than a float64, property keys are repeated on every single feature, and there is no index of any kind. It is a transport, not a store: land the output in GeoParquet or FlatGeobuf and delete the intermediate.

#Frequently Asked Questions

Why can’t I just stream a normal GeoJSON FeatureCollection?

Because a FeatureCollection is a single JSON value, and JSON has no framing: nothing in the byte stream tells a parser where one feature ends except the grammar itself. A conforming parser must therefore track nesting depth and string escaping across the whole document, and the standard libraries simply build the entire object in memory. Streaming parsers exist, but they push a state machine onto the consumer. One feature per line moves the framing into the format, where a newline is unambiguous.

Is newline-delimited GeoJSON a standard?

It is a widely implemented convention rather than a formal specification, sometimes labelled GeoJSONL, GeoJSON Text Sequences, or NDJSON of GeoJSON features. GDAL, tile builders, and most data platforms read and write it. The important rules are consistent across implementations: one complete JSON object per line, no wrapping FeatureCollection, no literal newlines inside a feature, and UTF-8 throughout.

Should line-delimited GeoJSON be a storage format or just an intermediate?

An intermediate. It fixes GeoJSON’s streaming problem and none of its other problems: coordinates are still decimal text, keys are still repeated on every feature, and there is still no index, so it remains several times larger than a compressed columnar file and cannot answer a spatial query without a full scan. Use it to move data between stages, and land the result in GeoParquet or FlatGeobuf.

#Related

• GeoJSON Overhead and Serialization Costs — parent guide: where GeoJSON’s bytes and CPU actually go
• Optimizing GeoJSON Payloads for APIs — shrinking what crosses the wire when GeoJSON is the required response
• Memory-Safe Chunked Writes for Large Shapefiles — the same bounded-memory discipline on the write side
• Converting GeoParquet to PMTiles with Tippecanoe — a build step that consumes exactly this stream

← Back to GeoJSON Overhead and Serialization Costs