fix(ingest): heartbeat + derive-stage watchdog so a hung stage can't wedge the run (#671)#673
Merged
Merged
Conversation
…wedge the run (#671) On a VPS (Linux, SurrealDB 3.2.0) an ingest run stalled for 300s+ after the subagents/claude-sidecars/invoked-positions stages with no further output. The pipeline is a single-pass Deferred DAG (no loop) - so this is a hang in a later `derive` stage whose SurrealDB query blocks on that server version, not a loop; the last-printed stages just looked like one. Two changes to runPipeline: - Heartbeat: every AX_INGEST_HEARTBEAT_SECONDS (default 30, 0 disables) log the set of stages still running, so a stall is attributable to a specific stage instead of a silent hang. Runs as a forked effect that `Effect.race` interrupts the instant the pipeline finishes or fails. - Derive-stage watchdog: cap `derive`-tagged stages at AX_STAGE_TIMEOUT_SECONDS (default 300, 0 disables). Past the cap the stage is failed OPEN - a warning + empty BaseStageStats sentinel - so downstream deps (which only await the Deferred and re-query the DB) proceed and the run completes and exits. Heavy ingest/provider stages (claude/codex/git) are EXEMPT: a full backfill legitimately runs for many minutes. Doesn't pinpoint the root SurrealDB-3.2.0 hang (not reproducible on 3.0.4), but turns "never exits" into "one derive stage skipped, run finished" + names it. Tests: env-parsing for both knobs; a hung derive stage fails open + downstream still runs + run resolves; an ingest stage past the cap is NOT killed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploying ax with
|
| Latest commit: |
956bac8
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://a4710fec.ax-62d.pages.dev |
| Branch Preview URL: | https://fix-671-ingest-stage-watchdo.ax-62d.pages.dev |
…ine hang)
Effect.race resolves on the first SUCCESS, so racing the pipeline against the
never-succeeding heartbeat made a FAILING pipeline hang until the caller's
timeout (broke ingest-workflow's "run_finished{failed}" test). Fork the
heartbeat as a background fiber and interrupt it via ensuring once the pipeline
settles, so the pipeline's success OR failure propagates unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Necmttn
added a commit
that referenced
this pull request
Jul 9, 2026
…locks ingest on SurrealDB 3.2.0) (#676) Root-caused #671 by reproducing on SurrealDB 3.2.0: the ingest does NOT hang or loop - it runs every stage and then CRASHES on the run-evidence derive stage: ingest: FAILED - Incorrect arguments for function record::id(). Argument 1 was the wrong type. Expected `record` but found `NONE` `command_outcome` and `hook_command_invocation` legitimately have NONE `tool_call` (session-level hooks: UserPromptSubmit/Stop/…; hook-less outcomes - 3,812 such rows in the repro), and `record::id(NONE)` throws. Both SQLs now guard it: `IF tool_call != NONE THEN record::id(tool_call) ELSE NONE END`. The crash aborts the whole ingest, so this is a strong candidate for the reporter's "never exits" (a killed/aborting run with no per-stage signal before #673's heartbeat). Also fixes a latent second bug found while verifying the read surface: `ax runs evidence` ordered by `ts` without selecting it, which SurrealDB rejects ("Missing order idiom `ts`"). It was masked on dev boxes where the run_evidence tables never existed. Now selects `ts`. Verified live end-to-end on a fresh SurrealDB 3.2.0 DB: run-evidence stage completes (written=55,631 refsWritten=18,882) and `ax runs evidence <id>` renders. Regression test asserts both tool_call SQLs are NONE-guarded (builders exported). Refs #671, #675. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #671.
Diagnosis
The reporter saw
ax ingeststall for 300s+ aftersubagents → claude-sidecars → invoked-positionson Linux + SurrealDB 3.2.0.runPipelineis a single-pass Deferred DAG — each stage runs exactly once, no loop — so this is a hang, not a loop (the last-printed stages just looked like one). Noderivestage has an unbounded internal loop either. The block is a laterderivestage whose SurrealDB query hangs on that server version;AX_INGEST_TIMEOUT_SECONDS(900s) does eventually fire, so it's not literally "never exits" — the reporter killed at 300s first.I can't reproduce the root hang (no Linux/SurrealDB-3.2.0 here). This PR implements the issue's explicit fallback ask — "report which derive stage is still producing edges" — plus makes a single hung derive non-fatal.
Changes (both in
runPipeline)AX_INGEST_HEARTBEAT_SECONDS(default 30,0disables) logs the set of stages still running, so a stall is attributable to a named stage instead of a silent hang. Forked;Effect.raceinterrupts it the instant the pipeline finishes/fails.derive-tagged stages atAX_STAGE_TIMEOUT_SECONDS(default 300,0disables). Past the cap the stage is failed open (warning + emptyBaseStageStatssentinel), so downstream deps — which only await the Deferred and re-query the DB — proceed and the run completes and exits. Heavy ingest/provider stages (claude/codex/git) are exempt: a full backfill legitimately runs for many minutes (my own dogfood ingest took >900s on the codex stage).Limits (honest)
Doesn't fix the underlying SurrealDB-3.2.0 query hang — turns "never exits" into "one derive stage skipped, run finished" and names the culprit so we can fix it once we have a repro. Pinpointing needs the reporter's full log / a Linux+3.2.0 repro.
Tests
bun test apps/axctl/src/ingest/stage/runner.test.ts→ 12 pass. New: env-parsing for both knobs; a hung derive stage fails open + downstream still runs + run resolves; an ingest stage running past the cap is NOT killed (exemption). Typecheck clean.🤖 Generated with Claude Code