Skip to content

[bulk] load succeeds when it reads nothing, so a wrong --format looks like an empty source #340

Description

@hunterhacker

Split out of #332's verification. #339 fixed the contradiction (an ERROR line above "Job completed successfully"); this is the behaviour question underneath it.

What happens

$ ./bulk load --table t --s3-path s3://bucket/data.json --format json     # the file is not JSON
WARNING - Read 0 items from 's3://bucket/data.json' as 'json' -- nothing was loaded. If that is unexpected, check --format and the path.
Job completed successfully. Job duration: 0:01:28
exit=0

Spark's JSON reader parses nothing out of the file and returns zero rows rather than raising, so load reports success. A script sees exit 0 and concludes the data landed.

Note the asymmetry that makes this worth fixing: --format parquet at the same file does raise (Parquet reads its footer eagerly), so it reports politely as the user's mistake — Could not read the source at '...' as 'parquet'. The same operator error is a clean failure in one format and a silent success in another.

Two cases that currently share one outcome

source today arguably
genuinely empty — no bytes, or a prefix with no matching objects success, "Read 0 items" success; an empty drop is a legitimate input, and the export pipeline treats a 0-item export the same way
holds bytes, yields 0 rows success, "Read 0 items" failure — the reader did not understand the data, which is a user-input error like any other

Proposal

Return the byte count from check_s3_file_exists (which already calls head_object, falling back to list_objects_v2 on a prefix, so the single-object case costs no extra API call) and branch on it:

The wrinkle to decide

A header-only CSV has bytes and zero data rows, and under this proposal it would fail. Options:

  1. Accept it and say so in the message (simplest, and a header-only CSV is usually a mistake too).
  2. Treat --format csv with --withHeader and exactly one line as empty. Costs a second read of the file and adds a special case.

I lean 1.

Related

The same "silently succeeded having done nothing" shape appears in #325 (a partial delete that aborts reports neither what it deleted nor what it skipped). Worth keeping the wording consistent across both when they are fixed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions