feat(hook-context-intelligence): ENOSPC disk circuit breaker + user-visible fail-loud alerts - #105
Open
Diego Colombo (colombod) wants to merge 1 commit into
Open
feat(hook-context-intelligence): ENOSPC disk circuit breaker + user-visible fail-loud alerts#105Diego Colombo (colombod) wants to merge 1 commit into
Diego Colombo (colombod) wants to merge 1 commit into
Conversation
…isible fail-loud alerts Stacked on #101 (atomic metadata writes + self-heal). Adds the resilience/UX layer on top: - ENOSPC/EDQUOT circuit breaker: skip disk writes for a capped exponential-backoff cooldown (5s..300s), then probe for recovery, instead of hammering a full disk on every event. Composes with #101 because its atomic writer re-raises OSError so this layer can classify ENOSPC. - Fail loud to the user via HookResult.user_message (bypasses the unwritable log file), severity matched to reality: PERMANENT DATA LOSS (error) when the event reached no sink (disk full and no destination / queue also full), a milder warning when still delivered to the server, and a one-shot info on recovery. - enqueue() now returns whether the event was queued so the handler distinguishes delivered from lost; network dispatch stays independent of disk state. Related: microsoft-amplifier/amplifier-support#492. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Independent of #101 — targets
main, no duplicated codeThis PR adds the disk-pressure resilience + user-visible alerting layer to
logging_handler.py. It is independent of #101 and targetsmaindirectly: it can merge in any order relative to #101 and shares none of #101's code (0 references to_atomic_write_text/_read_metadata— it only wraps methods that already exist onmain).bkrabach) fixes the corruption: atomic metadata writes + tolerant reads + self-heal of a 0-bytemetadata.json.The two are complementary and orthogonal. Related issue: microsoft-amplifier/amplifier-support#492 (this PR is related, not a full close — #101 is the corruption fix). Runaway
events.jsonlgrowth that filled the disk: design issue microsoft-amplifier/amplifier-support#493.What this adds (and delivers to users)
A full disk no longer spins or spams. An
ENOSPC/EDQUOTwrite failure opens a circuit breaker: disk writes are skipped for a capped exponential-backoff cooldown (5s → … → 300s), then one event probes for recovery — instead of retrying (and failing, and logging) on every event. Never gives up, never hammers.Disk-full fails loud, where the user can see it. When the disk is full the log file can't be written either, so
logger.warningis invisible. Alerts rideHookResult.user_message(surfaced in the UI), rate-limited, severity matched to reality:error: "PERMANENT DATA LOSS …"warning: local log stale, not lost.info: logging resumed.Events keep flowing to the server while the disk is down. Disk and network are independent sinks; the network fan-out runs regardless of disk state (
enqueueis a zero-I/O in-memory hot path).enqueue()now returns whether the event was queued, so the handler tells "delivered" from "lost" — the distinction that drives the alert severity above.Scope / relationship to #101
main: does not modify the metadata functions or add atomic-write/self-heal (that's fix: recover from corrupt metadata.json on disk full (ENOSPC) #101). No duplication.OSError, which this breaker classifies as ENOSPC; and fix: recover from corrupt metadata.json on disk full (ENOSPC) #101's self-heal repairs corrupt files this breaker's writes touch. Neither PR requires the other to be correct — together they resolve #492 end-to-end.logging_handler.pyinternal logic + one new test file only — no hook wiring, config, bundle-structure, or auth changes.Verification
main+ this layer (without fix: recover from corrupt metadata.json on disk full (ENOSPC) #101): 639 passed.test_logging_handler_disk_breaker.py(10 tests): breaker opens/skips/probes/recovers, capped backoff, non-ENOSPC does not trip it, dispatch stays independent of disk, and the three alert cases — no-destination + disk full = PERMANENT DATA LOSS (error), delivered-to-server = warning, queue-also-full = permanent loss.ruff check/ruff formatclean on changed files.Not covered here (deliberately)
events.jsonlgrowth — separate design issue #493.