[bulk] A denied or throttled batch write aborts the job with a raw Spark traceback, and leaves items behind silently

[hunterhacker]

Found while verifying #324 with a deliberately failing delete. Reproduction is cheap and exact (recipe at the bottom).

What happens

A DynamoDB resource-based policy denying BatchWriteItem for one partition key, then bulk delete over 60 items:

2026-08-28 22:20:35 ERROR TaskSetManager:267 - Task 161 in stage 4.0 failed 4 times; aborting job
exit=1
items remaining: 6            <- 54 of 60 were deleted

From the executor stream:

botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the
BatchWriteItem operation: User: arn:aws:sts::...:assumed-role/AWSGlueServiceRoleBulkDynamoDB-
DdbReadWrite-us-east-1/GlueJobRunnerSession is not authorized to perform ...

Three problems:

1. The user gets a Spark traceback, not an explanation. Nothing says "you are not allowed to delete from this table". The AccessDeniedException is in an executor stream, which the client discards (_pretty_print_log_event skips _g- streams), so the only thing on the console is Task 161 in stage 4.0 failed 4 times.
2. The partial delete is silent. 54 items were deleted and 6 were not. The driver's Deleted N items line never prints, because the job aborted first, so nothing states what did or did not happen. A user re-running blind is the best case; a user assuming nothing happened is the likely one.
3. find's per-item failure reporting cannot see it. batch_writer buffers 25 items and flushes, so the error is raised inside the with block's exit — outside the per-item try. Verified: zero Delete failed for lines were logged, and the M failed count added in [bulk] Bound per-item failure logging, and tell the user the total (closes #319) #324 stayed at zero. That count only covers malformed-record and key-extraction failures, which are rare; it does not cover the failure users will actually hit.

Why it is awkward to fix

A worker-side BulkExecutorError does not reach the driver as a BulkExecutorError — it arrives as a Py4JJavaError wrapping the worker traceback, so root.py's clean-error handler does not catch it, and the client's BulkExecutorError noise suppression never triggers either. Getting a clean message onto the console needs the driver to inspect the wrapped cause, not just a raise in the worker.

Options, roughly in increasing order of work:

1. Catch at the partition boundary and re-raise with a recognisable shape, then teach the driver (or root.py) to unwrap a Py4JJavaError whose cause carries our marker and print the underlying message. Fixes problem 1.
2. Report what was done before failing. Read the accumulator on the driver in a finally so Deleted N items, M not attempted prints even on an aborted run. Fixes problem 2.
3. Decide the policy for a wholesale write failure: abort (today) versus complete-and-report. Aborting is defensible for a permission error, but it must say so; completing with an accurate count risks a successful-looking exit 0 on a run that deleted nothing.

Reproduction

ddb.put_resource_policy(
    ResourceArn=f"arn:aws:dynamodb:us-east-1:{acct}:table/{T}",
    Policy=json.dumps({"Version": "2012-10-17", "Statement": [{
        "Sid": "DenyDeletingOneSpecificItem", "Effect": "Deny",
        "Principal": {"AWS": glue_job_role_arn},
        "Action": ["dynamodb:DeleteItem", "dynamodb:BatchWriteItem"],
        "Resource": f"arn:aws:dynamodb:us-east-1:{acct}:table/{T}",
        "Condition": {"ForAnyValue:StringEquals": {"dynamodb:LeadingKeys": ["k005"]}}}]}))

Then bulk delete --table <T> (PITR must be enabled first — the existing guard fires before any of this). Note the policy denies BatchWriteItem for a request containing the key, so one denied item poisons its whole 25-item batch, which is why 6 items rather than 1 survived: the failing task was retried 4 times and the job aborted while other tasks were still in flight.

Clean up with delete_resource_policy, then wait — DeleteTable returns ResourceInUseException until the policy update settles.

[hunterhacker]

Confirmed for the likelier case: a wholly read-only Glue role

@JZHunter asked whether this is only an item-level IAM problem. It is not — a read-only Glue job hits the identical mechanism, and worse.

Bootstrapped --XRole READ-ONLY, then bulk delete on a 10-item table with PITR enabled:

exit=1                              items remaining: 10      (nothing deleted)
[executor] AccessDeniedException ... when calling the BatchWriteItem operation
Job aborted due to stage failure: Task 155 in stage 4.0 failed 4 times

Same flush-time failure, same silence from the per-item reporter, but now every partition is denied, so nothing is deleted at all and the user has paid for a full job plus four retries per task. The AccessDeniedException text does eventually appear on the console — buried inside a GlueExceptionAnalysis JSON blob among hundreds of lines — but nothing says "this environment was bootstrapped read-only; delete needs write access."

This is the scenario worth optimising for: bootstrapping READ-ONLY for safety and then trying a mutating verb is a normal mistake, whereas an item-level deny is exotic.

The cheap fix this suggests: a pre-flight role check

Nothing reads the Glue job's role at verb time. But the client already calls glue_client.get_job() before every run (assert_version_parity), the response carries Job.Role, and the role type is already encoded in the name — constants.py has READ_ONLY_ROLE_ID = "DdbReadOnly" and READ_WRITE_ROLE_ID = "DdbReadWrite".

So a mutating verb (delete, fill, update, copy target, load) can refuse before starting anything:

This environment was bootstrapped read-only, so 'delete' cannot run.
Re-bootstrap with:  ./bulk bootstrap --XRole READ-WRITE

That turns a two-minute doomed job with a Spark traceback into an instant, actionable message, and it is independent of the harder work in this issue (unwrapping Py4JJavaError to surface worker-side causes). It does not cover custom roles, whose names we do not control — those still need the runtime path — but it covers the common case exactly.

Also found: the read-only role was missing three inline policies

Filed separately as #326. My first attempt at this test failed for an unrelated reason — the read-only role had no MinimalPricingAccess, so the job died in the cost estimate with AccessDeniedException ... GetProducts before reaching any write. Cause: bootstrap returns early when the role already exists unless _needs_role_refresh() fires, so a role created before those policies existed never gets them.

I added the three policies temporarily to make this test valid, then removed them, so that role is back as found. Probe table deleted, environment re-bootstrapped READ-WRITE, make test 1593 passed.

[hunterhacker]

The user-facing half is now split out as #327: a mutating verb on a read-only environment should say so in one sentence instead of printing 939 lines of Spark abort. That one is deliberately not a pre-flight check — per @JZHunter, whether a write will be permitted is not reliably knowable in advance — so it works by translating the failure into a BulkExecutorError, which root.py turns into the job's ErrorMessage and the client prints as its closing line.

This issue keeps the harder half: a failed write is never counted, because batch_writer raises at flush time outside the per-item try, so the M failed total added in #324 stays at zero for exactly the failures users hit. #327 can land first and independently.

[hunterhacker]

Most of this is fixed by #331; recording what is left so the issue is not read as wholly open.

Fixed

Problem 1 — the user gets a Spark traceback, not an explanation. find's delete_partition now catches broadly at the batch boundary and records through shared/worker_errors.py, and the driver surfaces it with raise_first_worker_error. A denial is classified as understood, so it exits through BulkExecutorError with no traceback. Measured live 2026-08-31 with a read-only role over a 24-item table:

46 lines total, 0 traceback/py4j/GlueExceptionAnalysis lines
SystemExit: Error during delete: User: arn:aws:sts::...:assumed-role/AWSGlueServiceRoleBulk331NoWrite/
GlueJobRunnerSession is not authorized to perform: dynamodb:BatchWriteItem on resource: ...

Problem 3 — find's per-item reporting cannot see a flush-time failure. Also addressed: because batch_writer buffers 25 items and flushes on with exit, the batch-level except added in #331 is what catches it. The failure is no longer invisible; per-item attribution still is not possible for a buffered flush, which is inherent to batch_writer.

The "why it is awkward to fix" section is obsolete. It assumed a worker-side BulkExecutorError cannot reach the driver as one, so root.py would need to unwrap a Py4JJavaError. #331 removed the need: workers never raise, they record on an accumulator and return, and the driver decides the shape. No Py4J unwrapping anywhere.

Still open — problem 2, the silent partial delete

54 of 60 items were deleted and the user is never told. #331 deliberately made a batch-level failure fatal before the count line, with this comment in find.py:

A batch-level failure means the delete did not do what was asked, so it is fatal rather than a count to report.

That is defensible for the exit code, but it means Deleted N items never prints, so the run says what went wrong and nothing about what it did.

The honest fix needs one more number than the code currently keeps. delete_partition tracks per-item failures (delete_failure_accumulator, #324) and the driver knows how many items matched, but nothing counts successful deletes — and when a batch flush is denied, the items that worker never attempted are not counted anywhere either. So it needs a success counter accumulator in delete_partition, read on the driver before raising, producing something like:

Deleted 54 of 60 matched items before the failure; the rest were not attempted.
Error during delete: User ... is not authorized to perform: dynamodb:BatchWriteItem ...

Small and self-contained, but it is a behaviour decision (report-then-fail) plus a new accumulator, so it is worth doing deliberately rather than folding into something else. The same gap exists for copy and fill, which report totals from accumulators that are only read on the success path.