fix(scanner): Consolidate scanner, MCP, and analysis behavior - #105
fix(scanner): Consolidate scanner, MCP, and analysis behavior#105reneleonhardt wants to merge 3 commits into
Conversation
Bundles four related behavior changes that touch the same scan paths, and
settles three contracts for how the scanner reports provenance:
- fix(mcp): Bound cancellable project scans — propagates request
cancellation through scanner, Git, handoff, and MCP traversals, and
bounds project and manifest discovery.
- feat(analysis): Add structured dependency contract — adds versioned,
deterministic dependency JSON and MCP structured output for agentic
coding in isolated worktrees and restricted sandboxes while preserving
the existing text response; coverage remains explicit, including known
partial Rust analysis.
- feat(scanner): Fail closed and report fallback provenance — prevents
failed, timed-out, or unavailable ast-grep scans from appearing
complete, and carries Cargo metadata fallback provenance through file
graphs.
- feat(scanner): Resolve JS/TS workspace imports — resolves local JS/TS
imports from package, pnpm, Bun, and Deno workspace manifests;
ambiguous, external, escaping, and unsupported targets remain
unresolved.
Three contracts this change settles:
1. One provenance vocabulary. analysis owns the shared contract
(SourceStatus/Source/Coverage/NormalizeCoverage); scanner consumes it
rather than defining a parallel copy.
2. One scanner entry point. ScanForDeps(ctx, root, filters)
(ScanOutcome, error) collapses the WithFilters x Context x Outcome
axes; all retired compatibility twins are removed.
3. Fail-closed without losing the degraded path. A timed-out or failed
ast-grep scan returns ScanOutcome{Analyses: nil, Sources:
[{ast-grep, timeout|failed}]} with a nil error — an honest, usable
answer; an unavailable scanner still returns ErrAstGrepNotFound, and
deps coverage derives from graph sources so the degraded status stays
observable.
CLI and MCP surface is unchanged: no new commands, arguments, or MCP
tools; existing output gains schema_version/coverage, JS/TS workspace
import resolution, degraded scan outcomes, and cancellation bounds.
Co-Authored-By: GPT-5.6 Sol <codex@openai.com>
… contracts Drop the filterAnalyses non-ctx wrapper so the two remaining callers use the ctx variant and propagate cancellation, and type GraphCoverage.Status as analysis.CoverageStatus so the provenance vocabulary stays single-sourced. Lock the settled contracts with a CoverageFromSources status matrix and a parser guard against the retired XxxContext/WithFilters/Outcome twins. Co-Authored-By: GPT-5.6 Sol <codex@openai.com>
NewDepsProject has no production callers — all use NewDepsProjectWithCoverage with graph-derived coverage. Unexport the default-coverage constructor, keeping it for its determinism and Rust-coverage tests. Co-Authored-By: GPT-5.6 Sol <codex@openai.com>
reneleonhardt
changed the title
There was a problem hiding this comment.
Pull request overview
This PR consolidates scanner, MCP, and analysis behavior around a single dependency-scan entry point, while introducing a shared provenance/coverage contract and improving cancellation + degraded-scan reporting across CLI/MCP/handoff paths.
Changes:
- Introduces
analysis-owned provenance + coverage contract (SchemaVersion,Coverage,Source,NormalizeCoverage) and threads it through scanner outputs and MCP structured responses. - Refactors scanner APIs to be context-aware (cancellation-propagating) and collapses retired
*Context/*WithFilterscompatibility twins into single entry points. - Adds fail-closed behavior for ast-grep scans and improves dependency graph fidelity (Rust cargo-metadata provenance + JS/TS workspace import resolution).
Reviewed changes
Copilot reviewed 45 out of 45 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| watch/watch_test.go | Removes intra-loop sleep in debounce test. |
| watch/daemon.go | Updates watcher daemon to call context-aware scanner APIs. |
| scanner/walker.go | Adds context-aware file/configured scans and cancellable analysis filtering. |
| scanner/walker_test.go | Updates tests for new ScanFiles(ctx, ...) signature. |
| scanner/types.go | Adds schema version + normalized coverage to deps JSON output. |
| scanner/rustgraph.go | Removes legacy GraphCoverage definition and keeps Rust coverage note constant. |
| scanner/rustcargo.go | Adds cargo-metadata fallback/mixed provenance calculation. |
| scanner/outcome.go | Introduces unified scan outcome + coverage derivation helpers and IncompleteScanError. |
| scanner/outcome_test.go | Adds tests for coverage/source outcomes and cargo-metadata provenance. |
| scanner/jsworkspace.go | Implements JS/TS workspace manifest parsing and local import resolution. |
| scanner/jsworkspace_test.go | Adds comprehensive workspace resolution + cancellation tests. |
| scanner/integration_more_test.go | Updates integration tests for new scanner entry points and degraded outcomes. |
| scanner/git.go | Makes git diff/stats/impact APIs context-aware and cancellation-friendly. |
| scanner/git_test.go | Updates git tests for new context-aware signatures. |
| scanner/filegraph.go | Refactors file graph build pipeline to consume scan outcomes, sources, cargo provenance, and JS workspace resolver. |
| scanner/filegraph_truth_test.go | Updates truth tests for new graph build APIs. |
| scanner/filegraph_test.go | Updates filegraph tests for new signatures and coverage types. |
| scanner/deps.go | Adds context-aware external dependency scanning with per-manifest byte budget support. |
| scanner/deps_test.go | Updates external dependency tests for new signature and error handling. |
| scanner/contracts_test.go | Adds tests for deterministic deps JSON + versioned coverage fields. |
| scanner/cancellation_test.go | Adds cancellation coverage across scanner, git, ast-grep, and deps reading. |
| scanner/bench_test.go | Updates benchmarks/tests for new ScanDirectory(ctx, ...) outcome shape. |
| scanner/astgrep.go | Adds fail-closed ast-grep scan outcomes with provenance + cancellation propagation. |
| scanner/astgrep_test.go | Adds tests for timeout/failure/unavailable scanner outcomes and new API usage. |
| scanner/api_surface_test.go | Guards against reintroducing retired compatibility twin APIs. |
| render/depgraph.go | Updates depgraph renderer to new scanner graph entry point. |
| mcp/main.go | Adds structured output schema for get_dependencies, enforces traversal root validation, and threads cancellation through MCP handlers. |
| mcp/cancellation_test.go | Adds end-to-end MCP handler cancellation/truncation behavior tests. |
| mcp/analysis_output.go | Normalizes coverage status typing for MCP outputs. |
| mcp/analysis_output_test.go | Verifies get_dependencies returns both text and structured content with schema + coverage. |
| main.go | Updates CLI to new scanner APIs; adds deps JSON schema/coverage and stdin deps graph building. |
| main_more_test.go | Updates CLI tests to expect coverage fields (including Rust partial coverage). |
| handoff/detail.go | Adds context-aware file detail dependency context resolution. |
| handoff/context_test.go | Adds cancellation + best-effort behavior tests for handoff context/detail. |
| handoff/build.go | Adds context-aware handoff build path, cancellable git ops, and cancellable hashing. |
| go.mod | Promotes jsonschema-go to a direct dependency. |
| docs/MCP.md | Documents MCP output/budget/cancellation semantics added by this PR. |
| cmd/intent.go | Updates intent coverage field to stringified CoverageStatus. |
| cmd/hooks.go | Updates hooks fallback hub scan to use context-aware graph build + configured scans. |
| cmd/context.go | Updates language detection and counting to use context-aware configured scans. |
| cmd/config.go | Updates project init scan to new ScanFiles(ctx, ...) signature. |
| blast_radius.go | Updates blast radius bundle construction to new scan outcome + context-aware APIs. |
| blast_radius_fixes_test.go | Updates parity test to compare provenance sources and new APIs. |
| analysis/contracts.go | Adds versioned analysis schema + provenance/coverage normalization contract. |
| analysis/contracts_test.go | Adds determinism/non-nil collection tests for coverage normalization. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Use BuildFileGraph for accurate file-level dependency resolution | ||
| fg, err := scanner.BuildFileGraph(project.Root) | ||
| fg, err := scanner.BuildFileGraph(context.Background(), project.Root, scanner.ConfiguredFilters(project.Root)) |
| graph, err = scanner.BuildFileGraphFromOutcome(context.Background(), absRoot, outcome, filters) | ||
| if err != nil { | ||
| fmt.Fprintf(os.Stderr, "Error building dependency graph: %v\n", err) | ||
| os.Exit(1) | ||
| } |
|
@codex review Focus areas for this one, since it consolidates four previously-separate PRs and touches every scan path:
Cross-platform behavior matters here: codemap ships on Windows, Linux, and macOS. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7fd5418fe9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| // Use BuildFileGraph for accurate file-level dependency resolution | ||
| fg, err := scanner.BuildFileGraph(project.Root) | ||
| fg, err := scanner.BuildFileGraph(context.Background(), project.Root, scanner.ConfiguredFilters(project.Root)) |
There was a problem hiding this comment.
When render.Depgraph is called from MCP handleGetDependencies, this line starts a second full dependency graph scan with context.Background() after the handler has already done a ctx-aware scan/build. If the client cancels while the textual response is being rendered, this ast-grep/filesystem work no longer observes the request context and can keep running until the scanner timeout; reuse the graph already built by the handler or thread the caller ctx into the renderer.
Useful? React with 👍 / 👎.
| if outcome.Status == ScanSourceAuthoritative { | ||
| return | ||
| } | ||
| c.Status = analysis.CoveragePartial |
There was a problem hiding this comment.
When ast-grep fail-closes on timeout or failure, BuildFileGraphFromOutcome still records that source with nil analyses, but this line turns every non-authoritative source into partial. In MCP importer/hub/file-context and blast-radius paths that read GraphCoverage directly, a timed-out primary scan is reported as partial even though no dependency references were extracted; timeout/failed-only coverage should remain unavailable, with only usable fallback/mixed sources promoting it to partial.
Useful? React with 👍 / 👎.
| targetString, ok := target.(string) | ||
| if !ok { | ||
| mappings.addInvalid(key) |
There was a problem hiding this comment.
For package-internal aliases such as "imports": {"#env": {"default": "./src/env.ts", "types": "./src/env.d.ts"}}, this parser marks the mapping invalid because it only accepts string targets. The resolver then drops #env imports from the graph even though the same conditional target shape is already handled for exports; package imports should use the same runtime-target selection instead of invalidating non-string mappings.
Useful? React with 👍 / 👎.
|
|
||
| if len(manifest.Files) == 0 { | ||
| return nil, nil, nil | ||
| return scanner.ScanOutcome{}, nil, nil, nil |
There was a problem hiding this comment.
With a valid --deps --stdin manifest whose files array is empty, this returns a nil graph and nil error, but runDepsMode later unconditionally reads graph.Coverage.Sources. That makes an empty manifest panic instead of producing the same empty dependency output as before; return an empty graph/coverage outcome or have the caller handle the nil graph case.
Useful? React with 👍 / 👎.
| return blastRadiusBundle{}, err | ||
| } | ||
| } | ||
| analyses := scanOutcome.Analyses |
There was a problem hiding this comment.
When ast-grep fail-closes on timeout/failure, ScanForDeps now returns nil analyses plus a degraded Sources entry with nil error, but this assignment immediately discards the sources and the blast-radius DepsProject/JSON never carries Coverage. In that scenario the deps and impact sections look like there simply were no changed dependencies, unlike the normal deps paths; keep the outcome sources and derive coverage for the blast-radius deps output too.
Useful? React with 👍 / 👎.
| return nil, err | ||
| } | ||
| if manifest := packages[root]; manifest != nil { | ||
| manifest.pkg.sourceRoot, manifest.pkg.outDir = parseTSOutputDirs(doc) |
There was a problem hiding this comment.
Packages whose tsconfig.json inherits rootDir/outDir via extends leave sourceRoot and outDir empty here, so exports like ./dist/index.js are never remapped back to src/index.ts and workspace package imports disappear from the graph. The older path-alias loader already follows extends; this workspace path should parse the merged config before calling parseTSOutputDirs.
Useful? React with 👍 / 👎.
| ExternalDeps: externalDeps, | ||
| DiffRef: diffRef, | ||
| } | ||
| depsProject := scanner.NewDepsProjectWithCoverage(absRoot, outcome.Analyses, externalDeps, diffRef, scanner.CoverageFromSources(graph.Coverage.Sources)) |
There was a problem hiding this comment.
This derives the degraded coverage correctly for codemap --deps, but the non-JSON path immediately passes the project to render.Depgraph, which never prints project.Coverage and, for fail-closed nil analyses, only says No source files found. A timed-out scan therefore looks like an empty complete graph to CLI users; include the coverage status/details in the rendered dependency output as well as in JSON.
Useful? React with 👍 / 👎.
What does this PR do?
Bundles four related behavior changes that touch the same scan paths, and settles three contracts for how the scanner reports provenance:
The three contracts this change settles
1. One provenance vocabulary
analysisowns the shared contract (analysis.SourceStatus,analysis.Source,analysis.Coverage,analysis.NormalizeCoverage); the scanner consumes it rather than defining a parallel copy.The byte-identical
scanner.ScanSourceStatus/ScanSourceOutcomeduplicate is gone — what remains are aliases (type ScanSourceStatus = analysis.SourceStatus,type ScanSourceOutcome = analysis.Source), so there is exactly one enum and one struct in the binary.Cargo-metadata and the new
rust-cargo(mixed) source feed the sameanalysis.Sourcelist.2. One scanner entry point
ScanForDeps(ctx, root, filters) (ScanOutcome, error)collapses the WithFilters × Context × Outcome axes.All retired compatibility twins were removed (
ScanForDepsContext,ScanForDepsWithFilters,ScanForDepsOutcome*,BuildFileGraphContext,BuildFileGraphWithFilters*,BuildFileGraphFrom*Context/FilteredAnalyses/OutcomeWithFilters,ScanDirectoryContext/Outcome,ScanConfiguredFilesContext,ScanFilesContext,ReadExternalDepsContext,GitDiffInfoContext,GitDiffFiles*,AnalyzeImpactContext,DependencyCoverageContext,HasConfiguredLanguageContext).codemaphas no external importers, so the compatibility surface was dropped rather than preserved; all callers (CLI, MCP, handoff, blast-radius, watch, render, cmd) were updated to the single API, and ctx is threaded through every subprocess-bounded path.3. Fail-closed without losing the degraded path
A timed-out or failed ast-grep scan now fails closed to an honest, usable answer instead of a hard error:
ScanOutcome{Analyses: nil, Sources: [{name: "ast-grep", status: timeout|failed, detail}]}is returned with a nil error, so hooks (which run codemap on every edit) and MCP keep a usable empty answer on exactly the large repos where a 30s timeout fires — previously that was a hard error.ErrAstGrepNotFoundwrapped inIncompleteScanError, so setup problems stay visible.--deps --json, MCPget_dependencies) is derived from the graph's source provenance viaCoverageFromSources, so a degraded scan is observable in the output (unavailable/partial) instead of silently looking complete; Rust repos reportpartialthrough therust-cargomixed source, and cargo-metadata fallback provenance is surfaced the same way.CLI / MCP surface
No new CLI commands or arguments, and no new MCP tools — the MCP tool set is unchanged (15 tools;
get_dependenciesmerely gains anOutputSchema).The changes affect the output and behavior of existing commands:
codemap --json --deps .output gainsschema_versionandcoverage(deterministic, versioned shape; MCPget_dependenciesreturns the same as structured content).codemap --deps <path>resolves local JS/TS imports from package, pnpm, Bun, and Deno workspace manifests.Sourcesprovenance instead of a hard error.Developed with carefully directed, manually reviewed AI assistance.
Co-Authored-By: GPT-5.6 Sol codex@openai.com