[bulk] Report driver-side failures politely too (closes #332) - #339
Merged
Conversation
added 6 commits
September 1, 2026 01:46
Workers have surfaced failures politely since #331; the driver had not. A denied connector read reached the user as ~320 lines: Glue's exception-analysis blob, Py4J's restatement, and a Java stack with AWS's actual sentence buried on line two, closing on "An error occurred while calling o304.load". Three parts. 1. root.py now routes every failure out of a verb through shared/driver_errors.py rather than re-raising. Understood -- a denial, expired credentials, throttling, or anything a verb already phrased as BulkExecutorError -- exits with one sentence. Unexpected prints its traceback once, for the user rather than for CloudWatch, and still exits with a one-line reason. Exiting rather than re-raising is the point: a re-raise is what summons Glue's blob and Py4J's duplicate. It catches BaseException, because a worker that calls exit() arrives as SystemExit through Py4J; driver_errors' own sys.exit is let through. 2. get_error_message now unwraps a Java/Py4J error. The SDK v1 pattern stopped matching when Glue moved to the v2 SDK (software.amazon.awssdk...DynamoDbException rather than com.amazonaws...AmazonDynamoDBException), so a denial's "message" was the whole stack. It takes the innermost cause, because the outer layer is usually Spark boilerplate ("Job aborted due to stage failure") while the cause is the sentence the user needs. 3. Per-site sentences where the net's generic one would be worse: sql's query errors, load's source read (naming the path and the format the user claimed) and its connector write, and the transform module named by --transform. The client suppresses Glue's blob on the driver banner as well as the worker one, and the drift guard now checks both constants against runner.py.
… every time
Found in the live before/after: a denied `find` printed our clean sentence and then 183
lines of Glue exception analysis anyway, while a denied `count` in the same batch printed
none. Glue emits that blob after a clean sys.exit only sometimes, and the client can only
drop it once the job has identified itself -- which previously happened only when the
exception was literally a BulkExecutorError.
Understood driver failures now log behind EXPLAINED_FAILURE_PREFIX ("Failure: "), the
client watches for it, and the drift guard checks all three markers rather than one.
Asserted client-side as well, since whether the blob shows up in any given live run is
not something a test can rely on.
…t that the client honours it
…notices it
The live before/after caught this: `--format parquet` at a CSV file was reported as an
unexpected failure ("Py4JJavaError: ... is not a Parquet file", with a traceback) rather
than as the user's own mistake. Parquet reads its footer while the DynamicFrame is being
created, so it raises from read_data -- outside the handler that already converted the
count() path, which is where a JSON or CSV mismatch surfaces.
read_data now sits inside that handler, so both report the same way: one sentence naming
the path and the format the user claimed it was.
Also updates the lint rule's invariant 2 for #332: the net exists, a marker has to reach
the console (Glue's blob follows a clean exit only sometimes -- 183 lines after a denied
find, none after a denied count in the same batch), and get_error_message has to unwrap
the Java stack to the innermost cause.
Measured while verifying #332: `sql` with a mistyped column produced 148 lines, 90 of them Spark logging the AnalysisException itself -- at ERROR, before our handler turns it into "SQL query error: [UNRESOLVED_COLUMN...]" -- as one 10,850-char JSON event carrying the message we already print, ~90 Java frames, and the unresolved query plan. One event, so one ignore-list anchor drops all of it, checked against the raw CloudWatch event rather than assumed. Anchored on the logger name inside the JSON; a guard test covers the tempting alternative of anchoring on the error text, which would have hidden the verb's own sentence too.
…or I committed fill/alwaysboom.py was a throwaway generator for reproducing a driver-side failure and should never have been committed -- removed. Coverage additions, all reachable now that there is a way to import shared/errors.py in a suite that stubs it: Spark's ParseException branch (message plus the offending SQL), the .message-attribute branch, and root.py's SystemExit passthrough. The first two were pre-existing gaps; the third is mine. shared/errors.py goes 54% -> 73%, root.py 96% -> 99%.
added 2 commits
September 1, 2026 15:25
…rip it The client matches markers as substrings anywhere in a log line, so "Failure: " was too generic: Spark emits shapes like "ExecutorLostFailure: executor 7 exited" and "FetchFailure: shuffle block missing", either of which contains it. A false match suppresses Glue's diagnostics for a failure nobody explained -- the opposite of what the marker is for. Across every run captured for #332 the string appeared only in our own output, so this is about the shapes we have not seen rather than one we have. Now "Bulk Executor failure: ", with a guard test over four real Spark and Glue lines; reverting the marker fails it. Re-verified live: a denied count is 31 lines, no traceback or Glue blob, closing on AWS's own sentence.
Reading zero items logged at ERROR and returned, so job.commit() ran, Glue marked the run SUCCEEDED, and the output read: ERROR - No data found, please check your data source Job completed successfully. Job duration: 0:01:28 An error and a success about the same run, with exit 0 for anything watching. Now a warning that says what happened and what to check, naming the path and the format the user claimed: "Read 0 items from '<path>' as 'json' -- nothing was loaded. If that is unexpected, check --format and the path." Deliberately still a success. An empty drop is a legitimate input and the export pipeline already treats a 0-item export that way. Whether a source that holds bytes but yields no rows should fail instead -- the wrong-format case -- is a behaviour change with its own before/after, tracked as #340.
This was referenced Sep 1, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #332.
A driver-side failure used to reach the user as three copies of the same problem: Glue's exception-analysis blob, Py4J's restatement, and a Java stack with AWS's actual sentence buried on line two. A denied
countwas 323 lines, closing onError Category: UNCLASSIFIED_ERROR; ...; An error occurred while calling o304.load— which names nothing anyone can act on.Workers have been polite since #331. This applies the same model to the driver.
Three parts
1. A net in
root.py. Everything out of a verb now goes throughshared/driver_errors.py, which classifies it with the sameclassify_failureworkers use and exits instead of re-raising. Re-raising is what summons the blob and the Py4J duplicate.BulkExecutorError) -> one sentence, no tracebackSystemExitpasses straight through: a helper that already calledexit()has said its piece.2.
get_error_messageunwraps a Java/Py4J error. The SDK v1 pattern in there stopped matching when Glue moved to the v2 SDK (software.amazon.awssdk...DynamoDbExceptionrather thancom.amazonaws...AmazonDynamoDBException), which is why a denial's "message" was the entire stack. It now takes the innermost cause, because the outer layer is usually Spark boilerplate ("Job aborted due to stage failure") while the cause is the sentence the user needs.3. Per-site wording where the net's generic sentence would be worse:
sql's query errors,load's source read (naming the path and the format the user claimed) and its connector write, and the module named by--transform.Plus two things the live runs turned up, described below.
Before / after, measured live
Same account, region, table (6 items) and
--XNumberOfWorkers 2;beforeismainat d5dcefa. "noise" counts lines matchingTraceback (most recent,py4j,GlueExceptionAnalysis,at org.apache.sparkorat software.amazon. Full logs for every row are in~/Documents/bulk-332-runs/with anINDEX.md.count, read denied (connector.load())find, read denied (connector.load())sql, read denied (connector.load())load, write denied (connector write)load --format parquetat a CSV filesqlreferencing a column that does not existnosuchcolumncannot be resolved. Did you mean one of the foll...fill, generator raises during the driver's size peekload-export --transformnaming a missing moduleload --format jsonat a non-JSON file (already benign)sqlwith a query the client rejects (already benign)countsuccessfind --limit 3successsqlsuccessloadsuccess (CSV)The
fillrow's single remaining noise line is the traceback we print on purpose — that failure is a bug in the user's own generator, so the frames are the report. It points atalwaysboom.py", line 5, in generate.Two things the live runs caught that unit tests had not
A marker has to reach the console every time. The first pass left a denied
findprinting our clean sentence and then 183 lines of Glue exception analysis anyway, while a deniedcountin the same batch printed none. Glue emits that blob after a cleansys.exitonly sometimes, and the client only suppresses it once the job has identified itself — which previously required the exception to literally be aBulkExecutorError. Understood failures now log behindEXPLAINED_FAILURE_PREFIX, and the drift guard checks all three markers againstrunner.pyrather than one.A format mismatch fails in two different places.
--format parquetat a CSV file raises while the DynamicFrame is created (Parquet reads its footer), not atcount()where a JSON or CSV mismatch surfaces — so it fell past the handler I had converted and was reported as unexpected (Py4JJavaError: ... is not a Parquet file).read_datanow sits inside that handler: 1006 lines -> 810, noise 104 -> 7.Known remaining noise, deliberately not suppressed
Two rows are still long, and none of it is our output. Taking
04_load_DENIED_write_connector(794 lines) line by line:[Glue Exception Analysis]JSON blobs from Glue's ownGlueExceptionAnalysisListener— one per failure level (GlueExceptionAnalysisTaskFailed,StageFailed,JobFailed, then a one-lineRoot Cause Analysis Result). Each carries a"Failure Reason"and a"Stack Trace"array of Glue connector frames (glue.spark.dynamodb.write.DynamoDBDataWriter.flushBuffer(DynamoDBDataWriter.scala:186)->.commit(:211)). ~740 lines totalBulkExecutorError: Error in writing to table: User: ... is not authorized to perform: dynamodb:BatchWriteItem ...They are JSON rather than Python tracebacks, which is why the noise counter only scores 7 for this run: it matches
at org.apache.spark-style frame lines, and these are"Declaring Class"fields inside a JSON array. The stack trace a reviewer sees in that file is Glue's, four times over, before our code has been told anything.Why the client cannot suppress it. The client drops Glue-analysis lines only once the job has identified itself, and that happens at line 798 — after all four blobs. The ordering is inherent: the write is distributed, so Glue's listener reacts to the task failing, then the stage, then the job, while the driver only learns of it when Spark finally raises on the write call.
This is also why the three connector read rows collapse to 37 lines while this one does not: a denied
.load()fails on the driver before any task exists, so there are no per-task blobs to arrive first.load --format parquetat a CSV file (810 lines) is the same shape as this row.The fix available, and why it is not here. Suppress
GlueExceptionAnalysisListeneroutput unconditionally rather than only after the marker. That is newly defensible: since this PR the driver always explains itself, so those blobs are always redundant. The cost is that for a failure we classify as unexpected — where the Python traceback is thin and Glue's root-cause analysis is the real diagnostic — you would lose it. It is a one-line ignore-list entry plus a test, and I would scope it to the"Event": "GlueExceptionAnalysis...Failedblobs while keeping the singleRoot Cause Analysis Resultline. Wanted a decision on it rather than folding it in.The other route, pre-flighting the write permission, is ruled out by earlier discussion: whether a write will be permitted is not reliably knowable in advance, so a good failure beats a guess.
Tests
make test: 1682 passed, 48 skipped. Coverage:shared/driver_errors.py100%,root.py99%,shared/errors.py54% -> 73% (its ParseException and.messagebranches were pre-existing gaps, testable once there was a way to import a module the suite stubs).Mutation-checked, each failing at least one test: re-raising in
root.py(the original bug), not printing the traceback for unexpected failures, treating everything as understood, not collapsing or not bounding the reason, removing the Java unwrapping, taking the outermost cause instead of the innermost, dropping the understood marker, the client not watching it, movingread_databack outside the handler, and both wrong anchors for the Spark-analysis filter.e2e on this branch:
connector7 passed,commands12 passed,security22 passed. Theroot.pychange is global, so the whole suite mattered here.Also in this PR
Spark logs a query-analysis failure itself, at ERROR, as one 10,850-char JSON event carrying the message we go on to print plus ~90 Java frames and the unresolved query plan — 90 of the 148 lines in the
sqlmistyped-column run. One event, verified against the raw CloudWatch record, so one ignore-list anchor drops all of it; a guard test covers the tempting alternative of anchoring on the error text, which would have hidden the verb's own sentence.The lint rule's invariant 2 is rewritten for the new shape, including the two things to check when reading driver-side code: that a marker reaches the console, and that the message survived the Java stack.
Follow-up: the understood-failure marker is now named, not generic
Review question: is
'Failure: 'specific enough for the client to key on? It matches as a substring anywhere in a log line, so no.Measured first: across every run captured for #332 — thousands of lines —
"Failure: "appeared only in our own output (8 occurrences, all ours). The near neighbours were Glue's"Failure Reason"JSON key and lowercase"stage failure:"/"most recent failure:", neither of which matches.But that is an argument about the shapes we happened to see. Spark emits
ExecutorLostFailure: executor 7 exited …andFetchFailure: shuffle block missing, and both contain"Failure: ". A false match makes the client suppress Glue's exception-analysis blob for a failure nobody explained — precisely inverting the marker's purpose, and #302 is an open issue about lost-executor errors, so those lines are not hypothetical here.The marker is now
"Bulk Executor failure: ", with a guard test that runs four real Spark and Glue lines through_pretty_print_log_eventand asserts none of them flips the suppression flag. Reverting the marker to the generic form fails three tests.Re-verified live with a denied
count: 31 lines, zero traceback/Py4J/Glue-blob lines, closing on AWS's own sentence, capture inafter/01b_count_DENIED_read_marker_recheck.log.