Problem
With multiLine=true (merged in #17), a malformed file with unmatched quotes causes Spark to silently swallow multiple lines into a single field. The result is fewer rows with bloated field values — no error raised, _corrupt_record not triggered.
This is the inverse of the #16 bug: previously corruption showed as extra rows, now it shows as missing rows with oversized fields.
Suggested approach
- After read, compare actual row count against a raw newline count heuristic
- Or flag any field value exceeding a configurable max length
- Or both, behind a
validate_multiline=True option
Acceptance criteria
Problem
With multiLine=true (merged in #17), a malformed file with unmatched quotes causes Spark to silently swallow multiple lines into a single field. The result is fewer rows with bloated field values — no error raised, _corrupt_record not triggered.
This is the inverse of the #16 bug: previously corruption showed as extra rows, now it shows as missing rows with oversized fields.
Suggested approach
validate_multiline=TrueoptionAcceptance criteria