Skip to content

Validate multiLine reads against runaway quote consumption #18

Description

@Iwan-Dyke

Problem

With multiLine=true (merged in #17), a malformed file with unmatched quotes causes Spark to silently swallow multiple lines into a single field. The result is fewer rows with bloated field values — no error raised, _corrupt_record not triggered.

This is the inverse of the #16 bug: previously corruption showed as extra rows, now it shows as missing rows with oversized fields.

Suggested approach

  • After read, compare actual row count against a raw newline count heuristic
  • Or flag any field value exceeding a configurable max length
  • Or both, behind a validate_multiline=True option

Acceptance criteria

  • A file with unmatched quotes produces a warning or error, not silent corruption
  • Validation does not break well-formed multiLine files
  • Behaviour is documented

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions