Split out of #332's verification. #339 fixed the contradiction (an ERROR line above "Job completed successfully"); this is the behaviour question underneath it.
What happens
$ ./bulk load --table t --s3-path s3://bucket/data.json --format json # the file is not JSON
WARNING - Read 0 items from 's3://bucket/data.json' as 'json' -- nothing was loaded. If that is unexpected, check --format and the path.
Job completed successfully. Job duration: 0:01:28
exit=0
Spark's JSON reader parses nothing out of the file and returns zero rows rather than raising, so load reports success. A script sees exit 0 and concludes the data landed.
Note the asymmetry that makes this worth fixing: --format parquet at the same file does raise (Parquet reads its footer eagerly), so it reports politely as the user's mistake — Could not read the source at '...' as 'parquet'. The same operator error is a clean failure in one format and a silent success in another.
Two cases that currently share one outcome
| source |
today |
arguably |
| genuinely empty — no bytes, or a prefix with no matching objects |
success, "Read 0 items" |
success; an empty drop is a legitimate input, and the export pipeline treats a 0-item export the same way |
| holds bytes, yields 0 rows |
success, "Read 0 items" |
failure — the reader did not understand the data, which is a user-input error like any other |
Proposal
Return the byte count from check_s3_file_exists (which already calls head_object, falling back to list_objects_v2 on a prefix, so the single-object case costs no extra API call) and branch on it:
The wrinkle to decide
A header-only CSV has bytes and zero data rows, and under this proposal it would fail. Options:
- Accept it and say so in the message (simplest, and a header-only CSV is usually a mistake too).
- Treat
--format csv with --withHeader and exactly one line as empty. Costs a second read of the file and adds a special case.
I lean 1.
Related
The same "silently succeeded having done nothing" shape appears in #325 (a partial delete that aborts reports neither what it deleted nor what it skipped). Worth keeping the wording consistent across both when they are fixed.
Split out of #332's verification. #339 fixed the contradiction (an ERROR line above "Job completed successfully"); this is the behaviour question underneath it.
What happens
Spark's JSON reader parses nothing out of the file and returns zero rows rather than raising, so
loadreports success. A script sees exit 0 and concludes the data landed.Note the asymmetry that makes this worth fixing:
--format parquetat the same file does raise (Parquet reads its footer eagerly), so it reports politely as the user's mistake —Could not read the source at '...' as 'parquet'. The same operator error is a clean failure in one format and a silent success in another.Two cases that currently share one outcome
Proposal
Return the byte count from
check_s3_file_exists(which already callshead_object, falling back tolist_objects_v2on a prefix, so the single-object case costs no extra API call) and branch on it:BulkExecutorError:Read 0 items from '<path>', but it holds <N> bytes -- the data is probably not '<format>'. Check --format.The wrinkle to decide
A header-only CSV has bytes and zero data rows, and under this proposal it would fail. Options:
--format csvwith--withHeaderand exactly one line as empty. Costs a second read of the file and adds a special case.I lean 1.
Related
The same "silently succeeded having done nothing" shape appears in #325 (a partial delete that aborts reports neither what it deleted nor what it skipped). Worth keeping the wording consistent across both when they are fixed.