feat(hook-context-intelligence): ENOSPC disk circuit breaker + user-visible fail-loud alerts (stacked on #101) - #102
Closed
Diego Colombo (colombod) wants to merge 1 commit into
Conversation
Diego Colombo (colombod)
force-pushed
the
fix/metadata-atomic-write-and-disk-breaker
branch
from
August 26, 2026 15:24
370ecb7 to
319377f
Compare
Diego Colombo (colombod)
changed the base branch from
main
to
fix/metadata-json-corruption-on-enospc
August 26, 2026 15:25
…isible fail-loud alerts Stacked on #101 (atomic metadata writes + self-heal). Adds the resilience/UX layer on top: - ENOSPC/EDQUOT circuit breaker: skip disk writes for a capped exponential-backoff cooldown (5s..300s), then probe for recovery, instead of hammering a full disk on every event. Composes with #101 because its atomic writer re-raises OSError so this layer can classify ENOSPC. - Fail loud to the user via HookResult.user_message (bypasses the unwritable log file), severity matched to reality: PERMANENT DATA LOSS (error) when the event reached no sink (disk full and no destination / queue also full), a milder warning when still delivered to the server, and a one-shot info on recovery. - enqueue() now returns whether the event was queued so the handler distinguishes delivered from lost; network dispatch stays independent of disk state. Related: microsoft-amplifier/amplifier-support#492. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Diego Colombo (colombod)
force-pushed
the
fix/metadata-atomic-write-and-disk-breaker
branch
from
August 27, 2026 09:15
319377f to
30d2fa9
Compare
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #101 — review that first
This PR is stacked on top of #101 (
bkrabach— "recover from corrupt metadata.json on disk full"). Its base branch isfix/metadata-json-corruption-on-enospc, so the diff shown here is only the additive hardening layer on top of Brian's fix. When #101 merges, GitHub auto-retargets this tomain._atomic_write_text) + tolerant reads (_read_metadata) + self-heal of corrupt/emptymetadata.json. Dogfooded against 4 real corrupt sessions.Related issue: microsoft-amplifier/amplifier-support#492. Follow-up design issue for the runaway
events.jsonlgrowth that filled the disk: microsoft-amplifier/amplifier-support#493.What this adds (and delivers to users)
A full disk no longer spins or spams. An
ENOSPC/EDQUOTwrite failure opens a circuit breaker: disk writes are skipped for a capped exponential-backoff cooldown (5s → … → 300s) instead of being retried on every event; one event then probes for recovery. It never gives up and never hammers a full filesystem. This composes with fix: recover from corrupt metadata.json on disk full (ENOSPC) #101 because Brian's_atomic_write_textre-raisesOSError, letting this layer classify ENOSPC.Disk-full fails loud, where the user can actually see it. When the disk is full the log file can't be written either, so a
logger.warningis invisible. Alerts now rideHookResult.user_message(surfaced in the UI), rate-limited, with wording matched to the real severity:error: "PERMANENT DATA LOSS …"warning: local log stale, not lost.info: logging resumed.Events keep flowing to the server while the disk is down. Disk and network are independent sinks; the network fan-out runs regardless of disk state (
enqueueis a zero-I/O in-memory hot path).enqueue()now returns whether the event was queued, so the handler can tell "delivered" from "lost" — which is exactly the distinction that decides the alert severity above.Scope
logging_handler.pyinternal logic only — no hook wiring, config, bundle-structure, or auth changes.Verification
test_logging_handler_disk_breaker.py(10 tests): breaker opens/skips/probes/recovers, capped backoff, non-ENOSPC does not trip it, dispatch stays independent of disk, and the three alert cases — no-destination + disk full = PERMANENT DATA LOSS (error), delivered-to-server = warning, queue-also-full = permanent loss.ruff check/ruff formatclean on changed files.Not covered here (deliberately)
events.jsonlgrowth (what filled the disk) — separate design issue #493.os.replace(used by fix: recover from corrupt metadata.json on disk full (ENOSPC) #101's writer) is atomic on Windows but fails if the destination is held open by another handle; nothing here holdsmetadata.jsonopen across writes.