feat: recursive documentation for monorepos (#162) - #439
Open
Brad Huffman (ppsplus-bradh) wants to merge 19 commits into
Open
feat: recursive documentation for monorepos (#162)#439Brad Huffman (ppsplus-bradh) wants to merge 19 commits into
Brad Huffman (ppsplus-bradh) wants to merge 19 commits into
Conversation
Adds a recursive documentation strategy so large monorepos can be documented as a tree of wikis: each subproject gets its own nested `<subproject>/openwiki/` sub-wiki documenting its subtree in depth, while the repository-root wiki links down to them and covers only cross-cutting concerns. Closes the request in langchain-ai#162. How it works: - Subprojects come from an `openwiki/workspaces.json` manifest, or from `--recursive` auto-detection of common workspace layouts (pnpm/npm/yarn, Cargo, Go, uv, Gradle, Maven, .NET solutions, Bazel), which writes a manifest for review then proceeds. Manifest presence auto-enables recursion; `--recursive=false` forces a single run. - An orchestrator runs the existing per-repo core once per subproject (backend rooted at the subproject so the docs-only write guard, snapshot, metadata, plan cleanup, and index-sync all scope for free), then writes a generated `openwiki/workspaces.md` aggregation index, then the root run last so its index-sync links the aggregation in. One model is resolved and reused across all runs; runs are sequential. - Git evidence and the incremental no-op check are scoped per subproject (subproject-relative diff), so `--update` regenerates only subprojects whose own subtree changed and skips the rest without a model call. Root git evidence excludes nested `**/openwiki`. - .NET solution detection coarsens per-project paths to product-area roots (`area/src/X`, `area/tests/X` -> `area`) to avoid one wiki per .csproj. - Backward compatible: with no manifest and no flag, behavior is unchanged. Documents the current limitation that updates do not cascade across subprojects (a shared-dependency change refreshes only that sub-wiki and the root, not its dependents) in README.md and the orchestrator JSDoc. Co-Authored-By: Claude <noreply@anthropic.com>
Makes `--recursive` keep the workspace manifest current as the repo evolves,
so a scheduled CI run picks up newly added projects and drops removed ones
without manual manifest edits.
- Discovery re-runs on every recursive run (deterministic, ~20ms even on a
400-project repo) and MERGES with the existing openwiki/workspaces.json
rather than only auto-detecting when no manifest exists.
- New manifest schema separates concerns: a managed `workspaces` list
(detection-owned, path-only, sorted, regenerated each run) and a
hand-authored `overrides` map keyed by path (goal / name / exclude /
include), so customization and auto-discovery never collide.
- Idempotent write: the manifest is serialized to canonical bytes (fixed key
order, sorted entries) and rewritten only when it actually changed, so an
unchanged repo produces no manifest diff and no spurious CI commit.
- Manual groupings are preserved: a referenced path detection cannot surface
whose directory still exists is kept and promoted to `include: true`
(carrying any goal/name); one whose directory is gone is pruned, with a
warning when it carried a hand-authored override.
- An `include` override that would overlap a detected workspace is dropped
with a warning instead of being persisted, preserving the "never write an
unresolvable manifest" invariant.
- Legacy flat manifests (workspaces:[{path,goal,name}]) are read and migrated
in-memory, then re-emitted in the new shape without losing goals.
Documents the manual-preservation and overlap-guard behavior in README.md.
Co-Authored-By: Claude <noreply@anthropic.com>
![corridor-security[bot]](https://avatars.githubusercontent.com/in/1177339?s=60&v=4){kind=small}
There was a problem hiding this comment.
The recursive monorepo aggregation writes openwiki/workspaces.md via writeFile() without checking whether the destination is a symlink; a malicious repo can commit that path as a symlink to an arbitrary file outside the repo, causing the write to follow it and overwrite that target with attacker-influenced content when a developer runs recursive OpenWiki.
…repo-docs # Conflicts: # test/code-mode.test.ts # test/prompt.test.ts
The recursive monorepo generator writes files inside the repository being documented (openwiki/workspaces.md, workspaces.json, .workspaces-state.json). Both the destination paths and their contents are attacker-influenced — the repo under documentation is untrusted input — and `writeFile` follows symlinks by default. A malicious repo could commit one of these paths as a symlink to a file outside the repo (e.g. ~/.bashrc); running recursive OpenWiki would then follow the link and overwrite that target with generated content. Adds `writeGeneratedFile`, which before writing: - refuses if the destination is a symlink (lstat, does not follow it), and - refuses if the resolved parent directory escapes the repo root (realpath), catching a symlinked ancestor directory. Routes all four generated-file writes through it (the aggregation index plus the manifest and state writers, which had the same exposure the reporter's finding did not enumerate). Fails loudly on a rejected path rather than writing through the link. Addresses the Corridor review finding on langchain-ai#439. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
This PR introduces an arbitrary file overwrite vulnerability via symlink following in src/code-mode.ts, where the GitHub Actions workflow file is written with an unguarded writeFile() call that will follow a symlink committed by a malicious repository. The safe writeGeneratedFile() helper introduced elsewhere in the PR (which checks for symlinks via lstat and validates parent directory containment) is not applied to this write or to the AGENTS.md/CLAUDE.md writes in the same file.
…urity) Second Corridor finding on langchain-ai#439: `ensureCodeModeRepoSetup` wrote .github/workflows/openwiki-update.yml and AGENTS.md/CLAUDE.md into the target repo with bare `writeFile`, which follows symlinks — the same CWE-59 class as the workspaces.md finding. A malicious repo could commit the workflow path as a symlink to e.g. ~/.ssh/authorized_keys and have it overwritten when a developer runs OpenWiki in code mode. Extracts the `writeGeneratedFile` guard (introduced for the workspaces writes) into a shared `src/safe-write.ts` — it has nothing monorepo-specific about it, and code-mode is a lower layer that should not depend on the workspaces module. Routes both code-mode write paths through it, so every file OpenWiki writes into an untrusted target repo now refuses a symlinked destination (lstat) or a parent that escapes the repo root (realpath). Moves the guard's tests to test/safe-write.test.ts and adds a workflow-file (repo-root) attack case. Co-Authored-By: Claude <noreply@anthropic.com>
Collaborator
|
Nice! Excited for this. |
Reframe the "no dependency cascade" note from a limitation-awaiting-a-fix into a by-design boundary. A subproject run is isolated to its own subtree (filesystem rooted at the subproject dir; git evidence scoped with `-- .`), so it cannot read a dependency's source. Re-running a dependent after an unrelated dependency change would have no new information to act on. Public API changes already surface in the dependent's own subtree diff and regenerate normally; only dependency internals (invisible to dependents) are skipped, which is the desired behavior. Closes #2. Co-Authored-By: Claude <noreply@anthropic.com>
…repo-docs # Conflicts: # src/agent/index.ts # src/cli.tsx # src/code-mode.ts # test/code-mode.test.ts
…to feat/recursive-monorepo-docs
…repo-docs # Conflicts: # src/agent/index.ts # src/cli.tsx # src/code-mode.ts
Contributor
Author
|
Colin Francis (@colifran) happy to take any feedback or make adjustments to approach based on consensus/discussion if you've had any. |
|
Really looking forward to this feature. Any chance this can be merged soon? |
Collaborator
|
Hey tobolleo - I have this on my radar and I owe Brad Huffman (@ppsplus-bradh) a review on it. I'll prioritize it for this week! |
Contributor
Author
|
I'll get latest merged in. |
Re-integrate the recursive monorepo documentation feature (langchain-ai#162) against upstream's rewritten generation architecture. Upstream refactors since the merge base removed the seams the feature originally attached to: the domain-directory reorg (adc03d6), the CLI split into src/cli/ (817b2a0), the prompt module split, and the resumable page-job lifecycle that replaced the monolithic agent run (4882ba3). Re-integration: - recursionRole guidance moved from createSystemPrompt (which no longer serves repository runs) into createRepositoryPlannerPrompt/createRepositoryPagePrompt via recursionRoleGuidance. - The orchestrator now drives each run through runOpenWikiAgent (model resolution is per-run and network-free) instead of the removed resolveRunModel/runOne/refreshChatGptTokensIfNeeded. - wikiGoalOverride threaded through beginRepositoryRun: a manifest-supplied brief wins, else the run root's own openwiki/INSTRUCTIONS.md. - Restored GitScope subtree scoping for the per-subproject update no-op check and planner evidence. git status/diff are repo-wide regardless of cwd, so without a pathspec every sibling change would regenerate every subproject; the committed-path no-op now treats an empty in-subtree diff as a legitimate skip for scoped runs while preserving upstream behavior for normal runs. - skipRepoSetup so the orchestrator owns code-mode repo setup once at the root rather than scaffolding a dead nested workflow per subproject. - Preserved the feature's symlink-safe generated-file writes (writeGeneratedFile) through the merged code-mode.ts, alongside upstream's new provider-env block. - Preserved the .workspaces-state.json exclusions from both wiki indexing and the content snapshot. CLI recursion wiring re-homed into src/cli/app/app.tsx and src/cli/runners.ts; the --recursive flag/help/parse landed in src/cli/commands.ts via rename detection. Tests ported to their relocated files and adapted to the new architecture; per-subproject isolation is covered by test/agent/subproject-noop-scope.test.ts. Co-Authored-By: Claude <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 6e74852 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
feat-level change; bump openwiki minor. Resolves the changeset-bot check on PR langchain-ai#439. Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
Author
|
Colin Francis (@colifran) - Main merged in. Let me know. I've been preoccupied with other things, and am happy to revisit/consider alternative approaches. TBH, I'm having to get myself reoriented on the idea as a whole. Running against a monorepo of my own at the moment.... If it necessitates changes, I'll ping here. |
Re-integrate the recursive monorepo documentation feature (langchain-ai#162) against upstream's latest page-coverage resumability wave (page fast-forwarding, per-page update windows, cursor coding-agent target, arch diagram). Conflicts and their resolution: - src/generation/repository-run.ts: upstream replaced the inline planner `changedPaths` computation with a new `pageUpdateWindows` mechanism, and added `fastForwardUnchangedRepositoryPageCoverage`. Both new functions called `getRepositoryChangedPaths` unscoped, which would reintroduce the cross-subproject cascade the feature exists to prevent (a subproject would regenerate on any sibling commit; a blocked fast-forward would force needless regeneration). Extracted `plannerEvidenceScope(role)` and threaded a `GitScope` through `getRepositoryPageUpdateWindows` and `fastForwardUnchangedRepositoryPageCoverage` so both honor the run's subtree scope. Kept upstream's reworded `actor` doc comment. The separate no-op scope (subproject-only; root stays repo-wide) is unchanged. Non-recursive runs pass an undefined scope, so behavior is identical to upstream. - src/ingestion/code-mode.ts: kept the feature's recursive-aware `${updateCommand}` in the generated workflow while adopting upstream's new `id: openwiki` / `continue-on-error: true` (consumed by the downstream `steps.openwiki.outcome` failure-propagation steps). Also re-applied the feature's `**/openwiki` add-paths entry (nested sub-wikis must be committed) to the dogfood workflow fixtures the merge regenerated from upstream (examples/openwiki-update.yml, .github/workflows/openwiki-update.yml), and updated the corresponding test expectation. Full suite: 3646 passed / 3 skipped / 0 failed. Co-Authored-By: Claude <noreply@anthropic.com>
5 tasks
The recursive monorepo docs feature guarantees no cross-subproject cascade, but `createRepositorySourceSnapshot` built a repo-wide source fingerprint from two unscoped inputs: `git status --porcelain` (repo-wide, repo-root-relative paths) and the global HEAD commit. On a subproject RESUME this let a sibling's commit flip the fingerprint, discard the interrupted plan, and replan a subproject that never changed (plus a rarer mid-run stability-check race). Fresh runs were already safe via the scoped `getUpdateNoopStatus`. This also confirms and fixes a second, functional defect: because status paths are repo-root-relative (`pkg/openwiki/x.md`) while the openwiki and ignore filters expect subtree-relative paths, a subproject's own generated `openwiki/` output was not excluded from the fingerprint. During a page- rewriting update the uncommitted self-churn flipped the fingerprint at finish, so `hasRepositorySourceChanged` reported true and forced the interrupted branch on every such run. Fix: thread an optional `GitScope` through `createRepositorySourceSnapshot` / `createRepositorySourceFingerprint`, gated to `recursionRole === "subproject"` only (root and single-repo runs stay repo-wide and byte-identical). For a subproject scope: - the status query gains a `-- .` pathspec and each returned path is re-based onto the subtree by stripping the `git rev-parse --show-prefix` prefix before the openwiki/ignore filters see it (empty prefix at the repo root, so unscoped output is unchanged) — this fixes the self-churn defect too; - HEAD becomes the subtree's last-touching commit (`git log -1 -- .`) instead of the global HEAD, so a sibling commit no longer moves it. That commit is a real, reachable ancestor, so the one value still serves as a valid `git diff <base>..HEAD -- .` baseline when it flows into the page manifest as `entry.gitHead` (a subtree tree hash would break that and is rejected). When no commit has touched the subtree yet (untracked-only or unborn branch) an `unborn:`-prefixed sentinel is returned so `gitHead` is omitted, matching the existing unborn-branch behavior. Migration needs no schema bump: a pre-upgrade `.run.json` triggers one conservative replan on first resume, and pre-upgrade manifest entries carry a real commit that stays a valid diff base until re-stamped. Tests: subproject-scoped fingerprint units (sibling commit / sibling dirty / own openwiki self-churn stay stable; own src change flips; gitHead equals the subtree last commit; subtree-commit is a valid scoped diff base; empty subtree omits gitHead), a byte-identical golden for unscoped/root scope, and integration resume tests (sibling commit preserves the plan, own change replans, and the unscoped baseline that shows the cascade the scope removes). Co-Authored-By: Claude <noreply@anthropic.com>
Adversarial review of the subtree-scoped fingerprint (5cabc55) found no HIGH/MEDIUM defects; these address its three LOW findings: - readSubtreeFingerprintHead now confirms HEAD does not resolve (via `rev-parse --verify HEAD`, which reads a ref to a SHA without loading commit objects) before returning an `unborn:` sentinel. Previously any `git log` failure was treated as unborn, so a failure on a *born* branch (e.g. a corrupt object DB with an intact HEAD symref) was masked instead of propagating as the correctness error it is — restoring parity with the whole-repo `rev-parse` path. - Strengthened the tautological "gitHead equals subtree last commit" test to advance the global HEAD with a sibling commit first, so global HEAD != the subtree commit; it now fails without the fix. - Added the missing mixed-baseline migration coverage: a global (non-subtree) commit stays a valid subtree-scoped `git diff base..HEAD -- .` base across several intervening sibling commits (the first-update-after-upgrade path), and an unborn-branch subproject snapshot omits gitHead without throwing. Full suite 3659 passed / 3 skipped. Co-Authored-By: Claude <noreply@anthropic.com>
Pick up 4 upstream fixes (unrecognized-language rejection langchain-ai#761, GitHub Copilot non-GPT-5 streaming langchain-ai#744, Bedrock maxTokens langchain-ai#743, docs langchain-ai#759). One conflict, in beginRepositoryRun (src/generation/repository-run.ts): upstream langchain-ai#761 now resolves and rejects an unrecognized language before touching the repository, right where the feature computes its skipRepoSetup guard and noopScope. Kept both — the language rejection runs first, then the recursive repo-setup guard and the subproject no-op scope. Full suite: 3680 passed / 3 skipped. Co-Authored-By: Claude <noreply@anthropic.com>
The monorepo root run now consults each subproject's sub-wiki
quickstart (enumerated from the workspaces manifest) as read-only
reference material, so the repo-wide overview stays consistent with
how each subproject describes its own scope, naming, and terminology.
This is a prompt-only change. Sub-wikis remain a read-only reference,
never an invalidation edge: plannerEvidenceScope("root") stays
root-excluding-nested and fingerprintScope("root") stays repo-wide, so
there is no sub-wiki-regen -> root-regen churn loop. Read access
already existed (docs-only confinement gates writes only).
Co-Authored-By: Claude <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a recursive documentation strategy so large monorepos can be documented as a tree of wikis instead of one flat wiki. Each subproject gets its own nested
<subproject>/openwiki/sub-wiki documenting its subtree in depth, while the repository-root wiki links down to them and covers only cross-cutting concerns.Closes the request in #162.
Motivation
For a large monorepo, a single wiki either drowns in detail or stays too shallow to be useful. #162 asked for OpenWiki to "run recursively so it supports monorepos/large systems that can't otherwise be documented in a single wiki." This implements exactly that: granular per-subproject docs, with the root wiki as a map that references them.
How it works
openwiki/workspaces.jsonmanifest at the repo root, or — with--recursiveand no manifest — from auto-detection of common workspace layouts (pnpm/npm/yarn workspaces, Cargo, Gogo.work, uv, Gradle, Maven, .NET.sln/.slnx, Bazel). Auto-detection writes a manifest for review, then proceeds. Manifest presence auto-enables recursion;--recursive=falseforces a single-repo run.workspaceslist (detection-owned, path-only, regenerated) plus a hand-authoredoverridesmap keyed by path (goal/name/exclude/include), so customization and auto-discovery never collide. Manual groupings detection can't surface are preserved as long as their directory exists..last-update.jsonmetadata, plan cleanup, and index-sync all scope to that subproject with no special-casing — then writes a generatedopenwiki/workspaces.mdaggregation index, then the root run last (so the root's index-sync links the aggregation in). One model is resolved and reused across all runs; runs are sequential.--update, a subproject regenerates only when files in its own subtree changed since it was last documented (its ownopenwiki/.last-update.json), skipping unchanged subprojects without a model call. The root wiki regenerates every run..sln/.slnxdetection coarsens per-project paths to product-area roots (area/src/X,area/tests/X→area) to avoid one wiki per.csproj, guarded so it never collapses to a whole-tree wiki.Backward compatibility
With no manifest and no
--recursiveflag, behavior is unchanged — the single-repo path is byte-for-byte identical. Legacy flat manifests are read and migrated in-memory, preserving hand-authored goals.No dependency cascade (by design)
Updates do not cascade across subprojects: a change to a shared subproject (e.g. a common kernel) refreshes only that sub-wiki and the root, not the sibling subprojects that depend on it. This is a deliberate design boundary, not a missing feature.
A subproject run is isolated to its own subtree — the filesystem tools are rooted at the subproject directory and its git evidence is scoped with a
-- .pathspec — so a run cannot read a dependency's source. Re-running a dependent after an unrelated dependency change would have no new information to incorporate and would just reproduce the same sub-wiki. And when a dependency's public API changes, the dependent's own call sites change with it, which already shows up in the dependent's own subtree diff and regenerates it normally; only changes to a dependency's internals (invisible to dependents) are skipped, which is the desired behavior.This supersedes the earlier "dependency-aware invalidation" follow-up (ppsplus-bradh#2), which was closed as by-design once the isolation model made clear that a cascade would only trigger runs that can't act on the trigger. The README's "no dependency cascade" note explains the same rationale.
Testing