#Mapping DBF Field Types to Arrow Schema

The safe way to convert a dBASE DBF table into an Arrow schema is to read the field descriptors first and let each field’s type code plus width and decimal count drive the target type — not the type code alone. The five codes you will meet in GIS attribute tables map as follows: C (character) to Arrow string, N (numeric) to int64 when it has zero decimals or to decimal128 when it has a fixed scale, F (float) to double, D (date) to date32, and L (logical) to a nullable boolean. The two traps that corrupt data silently are mapping wide N fields to double (which rounds off long parcel IDs and currency values) and ignoring that DBF truncates every field name to 10 characters. This page gives the full mapping table, a typed builder, and a round-trip check. It is part of Schema Mapping for Legacy to Modern Formats, where the broader legacy-to-columnar translation rules live.

#Quick-Reference: DBF to Arrow Type Map

#Reading Types Before Records

Every DBF file begins with a header of field descriptors, and each descriptor carries three facts that together determine the correct Arrow type: a one-character type code, a byte width, and a decimal count. The mistake that most converters make is keying only on the type code — mapping every N to double because “numeric means floating point.” It does not. A DBF N field is a fixed-width ASCII number, and its meaning depends entirely on the decimal count. N(18,0) is an 18-digit integer that must not go anywhere near a float64’s 15–16 significant digits; N(12,2) is a fixed-scale decimal that belongs in decimal128(12, 2); only when you genuinely have approximate data does F (or a narrow N) justify double. Reading the descriptors up front lets you build the schema deterministically before touching a single record, which is exactly the contract a downstream CI/CD validation gate will later assert against.

Numeric precision loss is the failure with the highest blast radius because it is invisible. When N(16,0) parcel identifiers pass through a double, values above 2^53 round to the nearest representable float and neighbouring parcels collapse onto the same ID. The join that depended on that ID then silently drops or duplicates rows downstream. The rule is mechanical: zero-decimal N fields go to int64 (or decimal128 if the width exceeds 18 digits), and any N with a positive decimal count goes to decimal128 with the DBF decimal count as the Arrow scale. Reserve double for F fields and for cases where you have confirmed the data is genuinely approximate and no exact key depends on it.

The 10-character name limit is the other legacy scar. DBF hard-caps field names at 10 bytes, so POPULATION_DENSITY was written to disk as something like POPULATIO, and two long names can collapse to the same truncated form. You cannot recover the intended names from the DBF alone — the information is gone — so a robust converter carries an external mapping from truncated to intended names and deduplicates any collisions with a stable suffix. This is closely related to how null values in spatial schema mapping are handled: both are cases where the legacy encoding discards information that your mapping layer must reconstruct or explicitly record as lost.

#Building the Arrow Schema

The builder below reads DBF descriptors with pyogrio, applies the width-and-decimal rules, and restores truncated names from a supplied mapping. It returns a pyarrow.Schema you can hand straight to a Parquet or GeoParquet writer.

# Requires: pyarrow>=14.0, pyogrio>=0.7
from __future__ import annotations

from dataclasses import dataclass

import pyarrow as pa
from pyogrio import read_info


@dataclass(frozen=True)
class DbfField:
    name: str      # possibly truncated to 10 chars
    type_code: str # 'C', 'N', 'F', 'D', 'L'
    width: int
    decimals: int


def dbf_to_arrow_type(field: DbfField) -> pa.DataType:
    """Map a DBF field to an Arrow type using width and decimal count."""
    code = field.type_code.upper()
    if code == "C":
        return pa.string()
    if code == "F":
        return pa.float64()
    if code == "D":
        return pa.date32()          # DBF D is YYYYMMDD, no time component
    if code == "L":
        return pa.bool_()           # nullable; blank char becomes null
    if code == "N":
        if field.decimals > 0:
            # Fixed-scale decimal — preserve exact currency/area values.
            precision = min(max(field.width, field.decimals + 1), 38)
            return pa.decimal128(precision, field.decimals)
        # Integer: int64 is exact to 18 digits; wider needs decimal128.
        return pa.int64() if field.width <= 18 else pa.decimal128(field.width, 0)
    raise ValueError(f"Unsupported DBF type code: {field.type_code!r}")


def build_arrow_schema(
    fields: list[DbfField],
    name_map: dict[str, str] | None = None,
) -> pa.Schema:
    """Build an Arrow schema, restoring truncated names and deduping collisions."""
    name_map = name_map or {}
    seen: dict[str, int] = {}
    arrow_fields: list[pa.Field] = []
    for f in fields:
        target = name_map.get(f.name, f.name)
        if target in seen:                       # collision after truncation
            seen[target] += 1
            target = f"{target}_{seen[target]}"
        else:
            seen[target] = 0
        arrow_fields.append(pa.field(target, dbf_to_arrow_type(f), nullable=True))
    return pa.schema(arrow_fields)


def descriptors_from_path(path: str) -> list[DbfField]:
    """Read DBF field descriptors via pyogrio without loading records."""
    info = read_info(path)
    return [
        DbfField(name=n, type_code=_code(dt), width=w, decimals=d)
        for n, dt, w, d in zip(
            info["fields"], info["dtypes"], info["field_widths"],
            info["field_precisions"],
        )
    ]


def _code(numpy_dtype: str) -> str:
    """Collapse a pyogrio dtype string to a DBF type code (illustrative)."""
    mapping = {"object": "C", "int64": "N", "float64": "F",
               "datetime64": "D", "bool": "L"}
    return mapping.get(numpy_dtype.rstrip("[ns]"), "C")

#Validation

Confirm the mapping preserves values by round-tripping a sample and comparing the wide numeric fields exactly. A cheap check reads the source with pyogrio and the converted Parquet with pyarrow, then asserts the high-risk columns match:

# Requires: pyarrow>=14.0, pyogrio>=0.7
import pyarrow.parquet as pq
from pyogrio import read_dataframe

def check_precision(dbf_path: str, parquet_path: str, id_col: str) -> bool:
    src = read_dataframe(dbf_path, columns=[id_col])[id_col].astype("int64")
    dst = pq.read_table(parquet_path, columns=[id_col])[id_col].to_pandas()
    return src.equals(dst.astype("int64"))

Expected outcomes: wide N(16,0) identifiers compared as int64 should match exactly with zero drift; a decimal128(18,2) currency column should round-trip to the cent with no floating-point residue. If a comparison fails on a numeric column, the field almost certainly passed through double somewhere — recheck that zero-decimal N fields resolved to int64 and not float64. Dates should compare equal as date32 calendar days, and L fields should show blanks as nulls rather than False.

#Edge Cases and Caveats

The N(20,0) overflow. A 20-digit integer exceeds both int64 (max ~9.2 × 10^18, 19 digits) and a naive double. Route anything wider than 18 digits to decimal128(width, 0), which holds up to 38 digits exactly, and confirm your query engine reads decimal128 — most modern engines do, but a few older readers coerce it back to double.

Encoding of C fields. DBF has no universal text encoding; a Latin-1 or code-page-1252 attribute table read as UTF-8 mangles accented place names. Detect or configure the source code page explicitly and decode to UTF-8 during conversion, because Arrow string is UTF-8 by definition.

Zero-filled dates. DBF D fields for unknown dates are often stored as eight spaces or 00000000 rather than null. Map both to null explicitly, or date32 parsing will raise on the invalid 00000000 and abort the whole file.

#Frequently Asked Questions

Why do my DBF numeric fields lose precision when converted to Arrow?

Because a DBF N field with zero decimals and a width above 15 digits cannot fit in a float64 without rounding, and many converters map N straight to double. A 16-digit parcel identifier or a currency value stored as N(18,2) will silently lose its low-order digits. Map wide integer N fields to int64 and fixed-scale N fields to Arrow decimal128 with the DBF decimal count as the scale.

How do I recover field names truncated to 10 characters?

You cannot recover them from the DBF alone — the format hard-limits field names to 10 bytes, so POPULATION_DENSITY was already written as POPULATIO or similar. Keep an external mapping from truncated to intended names and apply it during conversion, and deduplicate any collisions where two long names collapsed to the same 10 characters by appending a stable suffix.

How are DBF dates and booleans represented in Arrow?

A DBF D field is stored as an eight-character YYYYMMDD string with no time or zone, so it maps cleanly to Arrow date32. A DBF L logical field holds a single character (T, F, Y, N, or a space for unset), which maps to a nullable Arrow boolean where the space becomes null. Never map a D field to a timestamp type, because the source carries no time-of-day information.

#Related

• Schema Mapping for Legacy to Modern Formats — the broader legacy-to-columnar mapping workflow
• Handling Null Values in Spatial Schema Mapping — null and sentinel handling for the same tables
• Understanding Parquet Columnar Storage for GIS — how the target columnar types are stored
• CI/CD Validation Hooks for GeoParquet Conversion — enforcing the mapped schema as a contract
• Building Batch Conversion Pipelines with Python — where the schema builder plugs into a batch job

← Back to Schema Mapping for Legacy to Modern Formats