Skip to content

[bulk] Client prints a urllib3 traceback after stopping a job #353

Description

@hunterhacker

What happens

When bulk stops a Glue job because the logs said it was doomed, the run sometimes ends with
a traceback from our own process, after the closing line:

2026-09-01 17:30:17,521 WARNING - Job was stopped. Job duration: 0:02:47
Exception ignored in: <urllib3.response.HTTPResponse object at 0x109a12110>
Traceback (most recent call last):
  File ".../urllib3/response.py", line 1289, in close
    self._fp.close()
  File ".../http/client.py", line 432, in close
    super().close() # set "closed" flag
  File ".../http/client.py", line 445, in flush
    self.fp.flush()
ValueError: I/O operation on closed file.

Measured on run jr_30e6d16a... (a sql query stopped after an executor ran out of
memory). It is the tail end of the CloudWatch Live Tail event stream being garbage-collected
after event_stream.close(), so it is harmless -- but it is the last thing on screen, it is
a traceback, and it names files in our virtualenv. A user who has just been told their job
failed now has to decide whether this is part of the failure. It is not.

"Exception ignored in" means Python is reporting an exception raised inside __del__, which
cannot propagate, so nothing downstream is affected and the exit code is unchanged.

Fix

Close the live-tail stream deterministically rather than leaving it to the collector -- the
close path in _watch_log_group already calls event_stream.close() in some branches but
not on the unhealthy-shutdown return, which is exactly the path where this was seen.

Not reproducible on every stop, which suggests it depends on when the collector runs relative
to interpreter shutdown; a test would need to drive the close path rather than assert on
output.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions