close(): bound client close so an unclosable HTTP client cannot hang session cleanup - #82
Merged
Merged
Conversation
…shutdown Two unbounded awaits on the mount()-cleanup path: CopilotClientWrapper.close() awaited _owned_client.stop() with no ceiling (reached from provider.close() and both shared-client refcount paths), and cancel_emit_tasks() awaited an unbounded gather -- cancelling a task is a request, not a guarantee, so a task that swallows CancelledError wedges the drain. Bound both with asyncio.wait_for(asyncio.shield(...)) against a new sdk.close_timeout_seconds policy value (default 5.0s), warn and abandon on timeout, and clear _owned_client before the await so a wedged client is never retried. Adds contract clause sdk-protection:Subprocess:MUST:8.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Defect
Two unbounded awaits on the
mount()-cleanup path, either of which hangs Amplifier's session cleanup for the entire process:CopilotClientWrapper.close()—await self._owned_client.stop(), no ceiling.stop()tears down the SDK's ~500MB Electron subprocess; a wedged or unresponsive subprocess leaves that await pending forever. Three call sites reach it:provider.py:1511(GitHubCopilotProvider.close()), and both shared-client refcount paths in__init__.py(_acquire_shared_clientreplacing an unhealthy client,_release_shared_clientdropping the last reference).GitHubCopilotProvider.cancel_emit_tasks()—await asyncio.gather(*tasks_to_cancel, ...), no ceiling. Cancelling a task is a request, not a guarantee: a task that swallowsCancelledError(or is blocked in a shielded await) never completes and wedges the drain.provider.close()awaits this before touching the client, and__init__.py:578's cleanup calls it directly.This repo already treats unbounded awaits as a defect class elsewhere —
session.abort_timeout_seconds,session.disconnect_timeout_seconds,singleton.lock_timeout_secondsall exist for exactly this reason. Cleanup was the gap.Root cause
On
origin/main, commit bc8d6a9:The SDK's own graceful-shutdown bound (
_RUNTIME_SHUTDOWN_TIMEOUT_SECONDS, added v1.0.2 — see thepyproject.tomldependency note) is the SDK's promise, not this provider's, and it only coversstop(). It does not cover the emit drain, and it is not a guarantee this module can make across SDK versions.Change
New contract clause
sdk-protection:Subprocess:MUST:8— every await on the cleanup path is bounded bysdk.close_timeout_seconds(new, default 5.0s, added toSdkConfiginconfig/_sdk_protection.py, the module's policy source of truth).CopilotClientWrapper.close()— boundsstop(). Fixed at the wrapper because it is the single choke point all three call sites funnel through; one bound coversprovider.close()and both refcount paths, instead of three drifting copies.cancel_emit_tasks()— bounds the drain gather, and reports how many tasks were abandoned.shieldlets the operation finish even when the enclosing task is cancelled;wait_forcaps the wait.self._owned_clientis cleared before the await, so a client that timed out or raised is never retried on a secondclose(). Idempotency is preserved (test_close_idempotentpasses unchanged).CancelledErrorstill propagates exactly as onmain— it inheritsBaseException, so the existingexcept Exceptionnever caught it and still does not.contracts/sdk-protection.mdupdated: MUST-8 clause with rationale and implementation, config table row, traceability row, changelog entry.Gates
uv run pytest -q(exactly what CI runs) → 1563 passed, 12 deselected. Full suite green.make testequivalent (pytest tests/ -q --tb=short -m "not live") → same 1563 passed.ruff check .→ All checks passed (branch and base).ruff format --check .→ 21 files would be reformatted on this branch, 21 on unmodifiedorigin/main— identical. My edits toprovider.pyandsdk_adapter/client.pyare format-clean; the one pre-existing offender insidetests/test_client_lifecycle.py(line ~1262, untouched by this PR) was deliberately left alone rather than reformatted into diff noise.pyright amplifier_module_provider_github_copilot tests→ exit 0, 0 errors.New tests:
test_client_lifecycle.py::test_close_is_bounded_when_stop_never_returnsstop()returns in <2s, logs the WARNING, drops_owned_client, leavesis_healthy()False, does not raisetest_client_lifecycle.py::test_close_normal_stop_logs_no_warningtest_client_lifecycle.py::test_close_timeout_defaults_to_five_secondstest_provider_close.py::test_cancel_emit_tasks_is_boundedCancelledErrorcannot wedge the draintest_provider_close.py::test_cancel_emit_tasks_cooperative_logs_no_warningEvidence the tests actually catch the defect
Both new bounded tests were run against the unmodified
origin/mainclient.pyandprovider.py, each under a 25s wall-clock cap:Both against this branch:
Two independent 124s confirms these are two separate hangs, not one defect seen twice — which is why bounding only
stop()would have left the process hangable.Logs:
dtu-artifacts/sweep/prov-others/copilot-{regression-proof-BASE-clientstop,regression-proof-BASE-emitdrain,regression-proof-FIXED,full-FIXED,pyright-FIXED}.logNo live run
Deliberate: the live end-to-end proof for this defect class is carried by the sibling prov-anthropic lane, which exercises the same contract against a real provider session. Live-marked tests here are deselected in CI by design; the hang-vs-bounded reproduction above is stronger for these specific code paths than a live session that happens not to wedge.
Cross-references
recipes-7nj/recipes-8srmicrosoft/amplifier-module-provider-anthropicchat-completions(close(): bound client close so an unclosable HTTP client cannot hang session cleanup amplifier-module-provider-chat-completions#19),openai(close(): bound client close so an unclosable HTTP client cannot hang session cleanup amplifier-module-provider-openai#85),vllm(close(): bound client close so an unclosable HTTP client cannot hang session cleanup amplifier-module-provider-vllm#42),azure-openai(close(): bound client close so an unclosable HTTP client cannot hang session cleanup amplifier-module-provider-azure-openai#39).ollama,gemini, andopenai-chatgptneed no change — none of them awaits a client close at all.