Data Compression
  1. Home
  2. Geospatial Storage Fundamentals & Format Comparison
  3. Shapefile Limitations in Modern Data Stacks
  4. Handling DBF Encoding and Field Name Truncation
Section: Geospatial Storage Fundamentals & Format Comparison 15 min read

Handling DBF Encoding and Field Name Truncation

A shapefile’s attribute table is a dBASE III file from 1983, and it loses two things on the way in: field names longer than ten characters, and any text whose code page it failed to record. Both losses are silent, both are irreversible from the file alone, and both surface months later as a column nobody can identify or a street name rendered as Ru� de la Paix. This page belongs to shapefile limitations in modern data stacks and covers the recovery procedure specifically.

Quick Reference

Symptom Cause Fix Primary Use Case
PARCEL_REF where a long name was expected 10-byte name field in the DBF descriptor External name mapping, applied at conversion Any legacy migration
BUILDING_H and BUILDING_1 Two names truncated to the same ten characters Detect collision, resolve from source schema Layers with verbose schemas
Ru� or Rué in text values Wrong code page assumed at read time .cpg sidecar, then header byte, then detection Non-English attribute data
Names uppercased Writer normalisation, not the format Restore case from the mapping Cosmetic but breaks joins

Where the Two Losses Come From

The DBF field descriptor is a 32-byte fixed record. Eleven of those bytes hold the field name, ten usable plus a null terminator. There is no extension mechanism, no continuation record, and no place to store the original name — so parcel_reference_number becomes PARCEL_REF and the rest is gone. When two names collide after truncation, writers append a digit, which means the mapping from name to meaning now depends on the column order at export time.

The encoding problem has a different shape. Text in a DBF is bytes, and the file has two places to say which code page those bytes are in: an optional .cpg sidecar, and byte 29 of the header, the “language driver” identifier. Both are optional, both are frequently absent or wrong, and a reader that guesses wrong produces plausible-looking text with corrupted accented characters — which then propagates into every downstream system.

The two fixed-width fields where a shapefile loses information The upper half shows the thirty-two byte DBF field descriptor: eleven bytes for the name of which ten are usable, one byte for the type code, four reserved bytes, one byte for length, one for decimal count, and fourteen reserved. A long column name is shown being cut at the tenth character with the remainder discarded and nowhere to record it. The lower half shows the DBF header with byte twenty-nine labelled as the language driver, marked optional and frequently zero, alongside an optional .cpg sidecar file that when present is authoritative. DBF field descriptor — 32 fixed bytes, no extension mechanism name — 11 bytes 10 usable + null terminator type 1 byte reserved 4 bytes length 1 byte decimals 1 byte reserved 14 bytes parcel_ref erence_number — discarded, with nowhere in the file to record it Where the code page is (or is not) declared parcels.cpg sidecar authoritative when present — often absent DBF header byte 29 — language driver optional, frequently zero, sometimes simply wrong When both are missing there is no correct answer in the file — only a best guess scored against the data. A wrong guess produces readable-looking text, which is why the corruption survives review and reaches production.

Detecting the Encoding and Restoring Names

The procedure is: prefer what is declared, detect when nothing is, and refuse to guess where a wrong guess would be unrecoverable.

python
# Requires: pyogrio>=0.9, geopandas>=1.0  (Python 3.10+)
from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path

CANDIDATES = ("utf-8", "cp1252", "cp1251", "iso-8859-1", "iso-8859-2", "cp850", "cp437")

# Byte 29 of the DBF header; only the values seen in practice are listed.
LANGUAGE_DRIVER = {
    0x01: "cp437", 0x02: "cp850", 0x03: "cp1252", 0x57: "cp1252",
    0x58: "cp1252", 0x59: "cp1252", 0x64: "cp852", 0x65: "cp866",
    0xC8: "cp1250", 0xC9: "cp1251",
}


@dataclass(frozen=True)
class EncodingVerdict:
    encoding: str
    source: str            # "cpg" | "header" | "detected"
    confidence: float


def declared_encoding(dbf: Path) -> EncodingVerdict | None:
    """Return the encoding the dataset declares, if it declares one."""
    cpg = dbf.with_suffix(".cpg")
    if cpg.exists():
        label = cpg.read_text(encoding="ascii", errors="ignore").strip()
        if label:
            return EncodingVerdict(label.lower(), "cpg", 1.0)

    with open(dbf, "rb") as handle:
        header = handle.read(32)
    if len(header) < 30:
        raise ValueError(f"{dbf} is truncated — not a readable DBF")
    driver = header[29]
    if driver in LANGUAGE_DRIVER:
        return EncodingVerdict(LANGUAGE_DRIVER[driver], "header", 0.8)
    return None


def detect_encoding(dbf: Path, *, expected_alphabet: str) -> EncodingVerdict:
    """Score each candidate code page against the letters the language uses.

    Deliberately conservative: a low top score means the caller should ask a
    human rather than accept a plausible-looking decode.
    """
    with open(dbf, "rb") as handle:
        sample = handle.read(1 << 20)

    best, best_score = None, 0.0
    for candidate in CANDIDATES:
        try:
            text = sample.decode(candidate)
        except UnicodeDecodeError:
            continue
        letters = [c for c in text if c.isalpha()]
        if not letters:
            continue
        score = sum(1 for c in letters if c in expected_alphabet) / len(letters)
        if score > best_score:
            best, best_score = candidate, score

    if best is None or best_score < 0.9:
        raise RuntimeError(
            f"cannot confidently determine the encoding of {dbf} "
            f"(best {best} at {best_score:.2f}) — supply it explicitly"
        )
    return EncodingVerdict(best, "detected", best_score)


def restore_names(
    dbf_names: list[str], mapping: dict[str, str]
) -> dict[str, str]:
    """Map truncated DBF names back to full names, refusing ambiguity."""
    collisions = [n for n in dbf_names if n[-1].isdigit() and n[:-1] + "_" in mapping]
    if collisions:
        raise ValueError(
            f"truncation collisions need explicit resolution: {collisions}"
        )
    missing = [n for n in dbf_names if n not in mapping]
    if missing:
        raise KeyError(f"no full name recorded for {missing}")
    return {short: mapping[short] for short in dbf_names}

Validation

Two checks catch the two failures. The first proves the encoding by looking for replacement characters and for byte sequences that decode to implausible letters; the second proves that names round-tripped.

python
# Requires: geopandas>=1.0 — assert the conversion preserved text and names
import geopandas as gpd

frame = gpd.read_parquet("parcels.parquet")

# 1. No replacement characters anywhere in the text columns.
text_cols = [c for c in frame.columns if frame[c].dtype == object and c != "geometry"]
bad = {c: int(frame[c].astype(str).str.contains("�").sum()) for c in text_cols}
assert not any(bad.values()), f"replacement characters survived conversion: {bad}"

# 2. Every column carries its restored full name, not the ten-character stub.
stubs = [c for c in frame.columns if len(c) == 10 and c.isupper()]
assert not stubs, f"columns still carrying truncated DBF names: {stubs}"

Expected results: zero replacement characters, zero remaining ten-character uppercase names, and a column count that matches the source schema exactly. A count mismatch means a collision was resolved by dropping a column rather than renaming it.

The order to resolve a DBF encoding, and where to stop and ask A four-step decision chain. First, if a .cpg sidecar exists, use it and stop. Otherwise, if the header language driver byte is a known value, use its code page. Otherwise, decode a sample under each candidate code page and score how many letters belong to the expected alphabet. If the best score is at or above ninety per cent, accept it; if it is below, stop and require the encoding to be supplied by a human, because a plausible-looking wrong decode is worse than a failed job. .cpg sidecar? authoritative header byte 29? known driver value score candidates against expected alphabet ask a human best score < 0.90 no no low decode with the resolved code page and record which of the four routes produced it yes yes ≥ 0.90 Recording the route matters: a value that came from detection is a hypothesis, not a fact, and belongs in the dataset's metadata so a later reader knows how much to trust the text. Failing loudly beats producing mojibake that passes every downstream schema check. Handling a file that genuinely contains two encodings A decode chain applied per row rather than per file. Each value is first decoded as UTF-8; on failure it falls back to the declared code page, and on a second failure to a last-resort code page with replacement. The count of rows that needed each step is reported, so a file that is ninety-nine per cent UTF-8 with a legacy tail is handled without either failing the job or silently corrupting the tail. Decode per row, and count which step each row needed try UTF-8 4,118,204 rows fall back: declared cp 81,406 rows last resort + replace 272 rows The counts are the deliverable: 272 rows needed replacement characters and can be listed, inspected and fixed, rather than being discovered by a user months later. A file-wide decode would have had to choose between failing the job and corrupting 81,678 rows.

Edge Cases and Caveats

A .cpg that lies. Sidecars are written by hand as often as by software, and a file saying UTF-8 next to a cp1252 DBF is common. Treat the sidecar as authoritative but still run the scoring check as a validation, and log a warning when the two disagree rather than silently preferring one.

Names that differ only in case. Some writers uppercase all field names, so Area and area collide before truncation is even involved. Because target formats are case-sensitive, restoring the wrong case breaks joins downstream. Carry the case in the external mapping alongside the full name.

Mixed encodings inside one dataset. A layer assembled from several sources can genuinely contain both cp1252 and UTF-8 rows, and no single decode is correct. The honest handling is to decode per-row with a fallback chain, count how many rows needed the fallback, and surface that count — the same posture the pipeline takes toward null handling in schema mapping.

Frequently Asked Questions

Why are shapefile field names limited to ten characters?

The DBF field descriptor is a fixed 32-byte record whose name field is eleven bytes, ten of which hold characters and one of which is a null terminator. That layout comes from dBASE III in the early 1980s and cannot be extended without breaking every reader. So a column called parcel_reference_number becomes parcel_ref on the way in, and there is nowhere in the file to record what it used to be.

How do I find out what encoding a DBF file uses?

Check three places in order. A .cpg sidecar file next to the .dbf names the code page explicitly and is authoritative when present. Byte 29 of the DBF header holds a language driver identifier that maps to a code page, though many writers leave it zero. Failing both, decode a sample of the text columns under each plausible code page and score the results — real words in the expected language score high, and mojibake scores low.

What happens when two long field names truncate to the same ten characters?

Most writers append a digit, so building_height and building_heat_source become BUILDING_H and BUILDING_1. That silently destroys the association between name and meaning: nothing in the file says which is which, and the numbering depends on column order, so two exports of the same layer can assign them differently. Detect the collision explicitly against the source schema and resolve it from an external mapping rather than trusting the suffix.

← Back to Shapefile Limitations in Modern Data Stacks