ops fixes for v0.2.5: Pebble level sizing knobs, lift the write deadline through the middleware, tolerate tar rc 1 in snapshot.sh, compact phase liveness, Go runtime memory metrics - #50
Merged
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…esholds) COSIFT_PEBBLE_TARGET_FILE_MB, COSIFT_PEBBLE_LBASE_MB, COSIFT_PEBBLE_L0_COMPACTION_FILES and COSIFT_PEBBLE_L0_STOP_WRITES set the matching pebble.Options fields only when present; unset leaves the options byte-identical to before. With the 2 MiB default a full graph persist left 132k L0 files on the production store, the follow-up range compaction ran 10.5 h and a plain restart wedged on the L0 stop-writes threshold.
statusCapturingWriter and recordingWriter lacked Unwrap(), so http.NewResponseController(w).SetWriteDeadline returned ErrNotSupported at every call site and the 60 s WriteTimeout stayed in force for /admin/checkpoint, /admin/eval-quick, /admin/hnsw-compact?wait, /admin/frontier-clear and the /answer SSE stream (a 4m31s checkpoint returned an empty reply and failed the snapshot job). Both wrappers now unwrap and a shared liftWriteDeadline logs a failure once instead of discarding it.
A serve restart during the 40-minute tar made tar exit 1 under set -e and aborted the snapshot. Checkpoint SSTs are hard-linked and immutable, so rc 1 is harmless; rc >= 2 still aborts. snapshot-test.sh reproduces both with stubbed curl, gcloud and pigz.
… compact Steps 3 and 4 of compactLocked ran silently under the write lock; on the production graph the URL-index rebuild alone took ~5 minutes while /stats.hnsw_compact.phase stayed on "compact". Both steps now log start and end with elapsed time and report compact:url-index and compact:entry-point through the existing progress hook. Locking is unchanged.
runtime/metrics samples (no stop-the-world) exposed as a runtime object on /stats and as cosift_go_* series on /metrics: heap objects, heap live, heap goal, memory limit (-1 when unset), total mapped memory, GC cycles, goroutines. The 2026-09-06 GC wall (heap pinned at the GOMEMLIMIT goal, /search p50 150 ms -> 4 s) reads as heap_goal - heap_live -> 0.
andreimarinescu
force-pushed
the
andrei/v0.2.5-ops-fixes
branch
from
September 12, 2026 14:45
22db638 to
83f5aa6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five small operational fixes from the 2026-09-11/12 maintenance window on the production box (17M docs, 83M vectors,
560 GB Pebble store). One commit per fix.
COSIFT_PEBBLE_TARGET_FILE_MB(L0 target file size, doublingper level up to L6),
COSIFT_PEBBLE_LBASE_MB,COSIFT_PEBBLE_L0_COMPACTION_FILES,COSIFT_PEBBLE_L0_STOP_WRITES. Unsetleaves
pebble.Optionsexactly as before (verified againstEnsureDefaults). One log line at open when any is set.Background: with Pebble's 2 MiB
target_file_sizea full graph persist left 132k L0 files; the following rangecompaction rewrote 259 GiB over 10.5 h and a plain restart wedged on
L0StopWritesThreshold(12). Documented indocs/ENV.md.statusCapturingWriterandrecordingWritergainUnwrap() http.ResponseWriter,so
http.NewResponseController(w).SetWriteDeadlinereaches the connection through the middleware chain. The five callsites use a shared
liftWriteDeadlinethat logs the error once instead of discarding it. Symptom fixed: a 4m31s/admin/checkpointreturnedcurl: (52) Empty replyand the snapshot service failed. Test: httptest server withWriteTimeout200 ms and the real middleware; a handler that lifts the deadline and sleeps 500 ms delivers its body.Note:
/answerSSE streams are no longer cut at 60 s by the server (the edge still bounds them).--warning=no-file-changed; rc 1 ("file changed as we read it", harmless on ahard-linked checkpoint) continues, rc ≥ 2 aborts.
scripts/snapshot-test.shreproduces both with stubbedcurl/gcloud/pigz.compactLockedlog start/end withelapsed time and report
compact:url-index/compact:entry-pointthrough the existing progress hook, so/stats.hnsw_compact.phaseno longer sits oncompactfor minutes with no signal. Locking unchanged./statsgainsruntime(heap_objects_bytes,heap_live_bytes,heap_goal_bytes,mem_limit_bytes(−1 when unset),mem_total_bytes,gc_cycles,goroutines) and/metricsgains the matchingcosift_go_*series, read viaruntime/metrics(no stop-the-world). The GC wall of 2026-09-06 (heap pinned at theGOMEMLIMIT goal, p50 150 ms → 4 s) is visible as
heap_goal − heap_live → 0.Deploy notes: the Pebble knobs ship as a separate systemd drop-in and are enabled on a maintenance restart only; existing
SSTs are rewritten as compactions touch them.