csvtriage

Load messy CSVs, recover what you can, and report every decision.

csvtriage is the CSV reader that never silently fails. It loads files other parsers choke on (wrong encodings, odd delimiters, ragged rows, mixed line endings, HTML junk, compression), recovers what it can, and hands back a structured ReadReport describing every assumption, repair, and quarantined row. So you can trust the data, or know exactly why you can't.

The parsing engine is Polars; encoding detection uses chardet and mojibake repair uses ftfy. Every path (read, peek, stream, and the CLI) goes through the same engine, so a previewed or streamed chunk types identically to a full read.

Installation

pip install -e .          # from a clone (PyPI release pending)
pip install -e ".[dev]"   # with the test / lint / type-check toolchain

Optional extras: [parquet] (PyArrow, for Parquet I/O) and [pandas] (.to_pandas()). Python 3.10–3.13.

Quick start

import csvtriage as ct

# Auto-detects encoding, delimiter, quote char, header, and compression.
frame = ct.read("messy_data.csv")

print(frame.shape)            # (rows, columns)
print(frame.report.explain()) # what was detected and repaired, in plain English

# Clean common messiness; the original frame is left untouched.
cleaned = frame.clean()

# Hand off to Polars / pandas / Arrow, or write out.
df = cleaned.to_polars()
cleaned.write_parquet("clean_data.parquet")

Reading

# Auto-detect everything.
frame = ct.read("data.csv")

# Or force any of it.
frame = ct.read(
    "data.csv",
    encoding="latin-1",
    delimiter="\t",
    has_header=True,
    columns_as_string=["zip_code"],   # keep leading zeros
    on_bad_lines="skip",              # quarantine bad rows instead of raising
)

# Aggressive line-level recovery for genuinely broken files.
frame = ct.read("scraped.csv", robust=True)

read() first tries a fast strict parse. If that fails, or you pass robust=True, it falls back to line-level recovery: per-line encoding fallback, mojibake repair, HTML/markup filtering, ragged-row handling, and mixed/old-Mac line endings. Recovered structure is recorded as repairs; rows that still can't be parsed are quarantined, never silently dropped. With on_bad_lines="error" (the default) an unrecoverable row raises ParseError; with "skip" it loads the rest and records each quarantined row in frame.report.

The report

Every read attaches a ReadReport at frame.report:

r = frame.report
r.encoding, r.encoding_confidence   # e.g. ("cp1252", 0.99)
r.delimiter, r.quote_char, r.has_header
r.rows_read, r.columns
r.repairs                           # list[str] of what was changed
r.quarantined                       # list[QuarantinedRow(line_number, raw, reason)]
r.rows_quarantined, r.is_clean

r.explain()                         # human-readable narrative
r.to_dict(); r.to_json()            # machine-readable, for logging/alerting

Cleaning

Frame.clean() (a wrapper over csvtriage.quick_clean) applies deterministic transforms and records each on a copy of the report. It never prints and never mutates the original frame:

cleaned = frame.clean(
    strip_whitespace=True,
    standardize_nulls=True,     # NA / null / none / - / "" -> null
    drop_empty_rows=True,
    drop_duplicate_rows=True,
    fix_column_names=True,      # "First Name" -> "first_name"
    parse_dates=False,          # opt-in date parsing
    dayfirst=False,             # interpret ambiguous dates as European
)
for change in cleaned.report.repairs:
    print(change)

Preview, inspect, stream

preview = ct.peek("huge_file.csv", rows=1000)   # reads only a bounded prefix

summary = ct.info("huge_file.csv")              # no full load
print(f"{summary['size_mb']} MB, ~{summary['estimated_rows']:,} rows")
print(summary["encoding"], summary["delimiter"])

for chunk in ct.stream("huge_file.csv", chunk_size=50_000):
    process(chunk.to_polars())                  # each chunk is a Frame

Command line

csvtriage info data.csv [--json]            # encoding, delimiter, size, est. rows
csvtriage peek data.csv -n 20               # preview the first rows
csvtriage clean in.csv out.parquet          # read + quick_clean + write
csvtriage convert in.csv out.parquet        # read + write (.csv / .parquet)
csvtriage validate data.csv                 # exit non-zero if rows were quarantined

Exit codes are script-friendly: 0 on success, 1 when validate finds quarantined rows, 2 on a usage or runtime error. convert/clean accept --robust and --report text|json|none.

Error handling

Errors are the library's own (csvtriage.errors) and carry an actionable message plus the parameter to get past them:

try:
    frame = ct.read("data.csv")   # a row that recovery could not place
except ct.ParseError as exc:
    print(exc)
# Failed to parse 'data.csv': 1 row(s) could not be recovered. Pass
#   on_bad_lines='skip' to load the rest and inspect report.quarantined.

The hierarchy: CSVTriageError (base), EncodingError (raised for an encoding name Python has no codec for), and ParseError. A missing file raises the builtin FileNotFoundError.

Performance

The strict path is Polars with a thin detection layer in front, so clean files read at close to raw pl.read_csv speed. Robust mode does extra per-line work and is slower by design; that is the cost of recovering files a strict parser rejects. No benchmark numbers are published here yet; a reproducible harness is on the roadmap rather than hand-quoted figures.

Examples

Runnable scripts in examples/:

• basic_usage.py: read a file and inspect the report
• encoding_recovery_example.py: non-UTF-8 input, auto-detected
• robust_reading_example.py: recover a contaminated file and review quarantine
• streaming_example.py: info, peek, and stream on a large file

Status

Alpha (v0.1.0). The core (detection, strict + robust reading, the report, cleaning, peek/info/stream, and the CLI) is implemented and tested across Python 3.10–3.13. On the roadmap: numeric/date normalization, recovery policies, CleverCSV-grade dialect detection, and optional validation.

Contributing

Issues and PRs welcome. Run the full gate before submitting:

make check    # black --check, ruff, mypy, pytest

License

MIT