fix(doctor): multi_source_drift completes on real brains — prune + wire env levers + raise file cap (SCA-3773)#3
Conversation
…re env levers + raise file cap (SCA-3773) The `multi_source_drift` doctor check walks each non-default source's filesystem to find pages misrouted to `default`. On any brain of normal size it hit its walk budget and reported "check skipped, walk too large", turning drift detection into a silent blind spot that read as healthy. The warning told operators to tune GBRAIN_DRIFT_LIMIT / GBRAIN_DRIFT_TIMEOUT_MS, but those env vars were never read — a dead-end lever. Root causes (both fixed): 1. File-count cap too low. A brain ingesting transcript/learning history blows past 10K markdown files in a single source (the brain that surfaced this had 13.5K transcript files in one source). The walk is fast — 13.5K files cross-checked in <300ms — so time was never the binding constraint; the count cap was. Raised 10K -> 50K. The 15s timeout (up from 5s) remains the real hang-guard. 2. node_modules/vendor/dist never pruned. When a source local_path points at a working repo, the dotfile-only skip missed these (none are dot-prefixed), so the walk descended dependency trees sync never indexes. The walk now applies the canonical pruneDir gate from sync.ts, so it considers exactly the files sync ingests — both a perf fix (scaffolde-ai source walk ~7.8s -> ~0.14s) and a correctness alignment. Also: - GBRAIN_DRIFT_LIMIT / GBRAIN_DRIFT_TIMEOUT_MS are now honored via resolveDriftLimit / resolveDriftTimeoutMs (explicit opts still win for tests; invalid values fall back with a once-per-process stderr warning, mirroring _resolveSyncFreshnessHours). - Walk drops the per-entry lstatSync in favor of readdirSync withFileTypes (symlink-following semantics unchanged). Verified end-to-end: live `gbrain doctor` now reports `multi_source_drift: ok — No cross-source slug drift detected` (was: `warn — skipped, FS walk hit limit/timeout`). 10/10 unit tests pass; doctor.test.ts 83/83. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c38d97f1e9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| }, | ||
| "license": "MIT", | ||
| "version": "0.42.51.0" | ||
| "version": "0.42.52.0" |
There was a problem hiding this comment.
This release bump updates package.json to 0.42.52.0 but leaves the tracked VERSION file at 0.42.51.0. The release instructions treat VERSION and the changelog as branch-scoped release metadata, so any release tooling or operator check that reads VERSION will still see the previous release while the package/changelog advertise 0.42.52.0. Please update the VERSION file in the same commit as the package bump.
Useful? React with 👍 / 👎.
| // Explicit opts (tests) win; otherwise honor the documented env-var levers | ||
| // GBRAIN_DRIFT_LIMIT / GBRAIN_DRIFT_TIMEOUT_MS (pre-SCA-3773 these were named | ||
| // in the doctor warning but never actually read — the lever was a dead end). | ||
| const limit = opts.limit ?? resolveDriftLimit(); |
There was a problem hiding this comment.
When an operator raises GBRAIN_DRIFT_LIMIT above ~65k to handle a source larger than the new 50k default, this value is used unchanged and the later batchProbeExistence call builds one SQL statement with one bind parameter per slug plus the source id. On Postgres brains that exceeds the 65,535-parameter limit, so the doctor check throws and the surrounding best-effort catch drops multi_source_drift instead of reporting drift. Please clamp this env value below the bind ceiling or chunk the DB probe.
Useful? React with 👍 / 👎.
| if (entry.isDirectory()) { | ||
| // Apply the canonical sync prune gate so the drift walk considers | ||
| // exactly the files the sync walker indexes: skip `node_modules`, | ||
| // `vendor`, `dist`, `build`, `venv`, `ops`, `.raw`, dot-dirs, and git | ||
| // submodules. This is both the perf fix (these trees dominate walk | ||
| // time on a working-repo source) and a correctness alignment — a file | ||
| // sync never ingests can't be a real drift candidate (SCA-3773). | ||
| if (!pruneDir(name, d)) continue; |
There was a problem hiding this comment.
On filesystems where readdir(..., { withFileTypes: true }) cannot provide a directory type (for example some network/FUSE mounts), Dirent.isDirectory() is false even for real directories. The old lstatSync(full).isDirectory() path still recursed in that environment, but this change treats those directories as ordinary non-markdown entries and misses all nested pages, making multi_source_drift report clean on mounted brains with people/*.md-style layouts. Please fall back to lstatSync when the dirent type is not definitive.
Useful? React with 👍 / 👎.
1 participant
Problem (SCA-3773)
gbrain doctor'smulti_source_driftcheck walks each non-default source's filesystem to find pages misrouted todefault. On a brain of normal size it hit its walk budget and reported "check skipped, walk too large" — drift detection became a silent blind spot that reads as healthy. The warning told operators to tuneGBRAIN_DRIFT_LIMIT/GBRAIN_DRIFT_TIMEOUT_MS, but those env vars were never read — the documented fix did nothing.Root causes (both fixed)
node_modules/vendor/distnever pruned. When a sourcelocal_pathpoints at a working repo, the dotfile-only skip missed these (none are dot-prefixed), so the walk descended dependency trees sync never indexes. The walk now applies the canonicalpruneDirgate fromsync.ts, so it considers exactly the files sync ingests — a perf fix (thescaffolde-aisource walk dropped from ~7.8s → ~0.14s) and a correctness alignment.Also
GBRAIN_DRIFT_LIMIT/GBRAIN_DRIFT_TIMEOUT_MSare now honored viaresolveDriftLimit/resolveDriftTimeoutMs(explicit call-siteoptsstill win for tests; invalid/non-positive values fall back with a once-per-process stderr warning, mirroringsync_freshness's_resolveSyncFreshnessHours).lstatSyncforreaddirSync({ withFileTypes: true })— symlink-following semantics unchanged (Dirent.isDirectory()does not follow symlinks).Acceptance criteria
multi_source_driftcompletes (not skipped) in a normalgbrain healthrunVerification
gbrain doctorfrom this branch against the affected brain now reportsmulti_source_drift: ok — No cross-source slug drift detected.(was:warn — skipped, FS walk hit limit/timeout). Combined walk of all 3 sources completes in ~286ms.test/multi-source-drift.test.ts10/10 pass (added: node_modules/vendor/dist pruned-not-counted;GBRAIN_DRIFT_LIMIThonored when opts omit it; resolver parse + garbage-fallback).test/doctor.test.ts83/83 pass. Fulltsc --noEmitadds zero new errors (5 pre-existing in unrelated files).🤖 Generated with Claude Code