feat(cobol): FILLER filter + callBound tier pinning + NIST dogfood baseline#52
Merged
Conversation
NIST CCVS dogfood (#52 follow-up) surfaced cross-copybook FILLER ↔ FILLER pairs as inferred-high noise — FILLER is COBOL's name-for- unnamed and a record may declare any number of unrelated FILLERs. Skip at the name-keyed candidate-sourcing layer. Scope: name-keyed pool only (high / ambiguous). The semantic tier builds its own pool and drops FILLERs separately — FILLERs without USAGE drop via the existing PIC+USAGE filter, FILLERs with USAGE drop via the same-name leaf gate in the semantic matcher. Ordering: checked pre-projection; a hypothetical `REPLACING FILLER BY ...` clause that renames FILLER into a named identifier is still discarded (laundering placeholder evidence — same precision-first stance as the Phase C leaf-rename gate at the candidate-sourcing site). Unit-test anchor in field-lineage.test.ts pins both halves: CUSTOMER-ID / CUSTOMER-NAME pair (recall intact), FILLER pair suppressed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same shape as PR #51's deterministic linkage pinning, applied to the `callBound` family. - `ExpectedCallBound.confidence?: "deterministic" | "high"` — optional, family-wide all-or-nothing pinning (matches the existing qualified-vs-leaf pinning rule on the same family). - `validateCallBound` accepts the new field and rejects invalid values + mixed-pin manifests with a path-pointing error. - `gradeCallBound` and `callBoundKey` plumb the optional confidence through both expected and actual key construction. When pinned, the canonical key gains a `|tier=<value>` suffix on both sides. - Basic fixture's callBound entry now pins `confidence: deterministic`. Test additions: - 2 validation cases (invalid value + mixed-pin rejection). - A wrong-tier regression anchor: clones the basic fixture's COBOL sources, rewrites the manifest with `confidence: high`, asserts every expected entry becomes FN and every actual becomes FP. A tier-agnostic harness would have scored 1.0/1.0. Closes a parallel precision gap to deterministic-linkage pinning: a builder regression that emits the correct call-bound pair under the wrong confidence tier (e.g. a `deterministic` shape-match landing as `high`, losing pictureMatch evidence) is now caught. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per-fixture eval-harness tests catch "did the manifest's pinned facts
emit," but not "did the builder start (or stop) emitting something the
manifests don't pin." The corpus-level baseline closes that gap.
Setup:
- `src/cobol/__tests__/corpora/ccvs-replacing-subset/` — curated
slice of the NIST CCVS COBOL85 test corpus (`newcob.val`,
public-domain). All SM* ("Source Manipulation") programs plus all
K-prefixed copybooks from the suite, ~780KB across 64 files.
Permissive copybook inclusion so every COPY directive resolves.
- `src/cobol/dogfood-lineage.ts` — library that parses every COBOL
file in a directory, runs `buildFieldLineage`, renders the
inferred-* emission as a sorted canonical-key text blob.
- `scripts/dogfood-lineage.ts` — CLI wrapper for regenerating the
baseline when behavior intentionally changes.
- `src/cobol/__tests__/corpora/ccvs-replacing-subset/baseline-inferred-high.txt`
— committed expected output. Today emits 4 inferred-high pairs
(all KEY-{2,3,4,5} between K501A and K501B), validated by the
PR #51 Phase C dogfood.
- `src/cobol/__tests__/dogfood-ccvs.test.ts` — vitest that runs the
dogfood and asserts the rendered output exactly matches the
committed baseline, with a sanity check that the baseline floor is
non-zero (catches a "all matches silently dropped + stale
baseline" both-empty pass).
`.gitattributes` locks LF on text + source + baseline file extensions
so Windows clones with `core.autocrlf=true` don't break the
byte-exact baseline comparison.
Doc updated with the regeneration procedure.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
xinhuagu
added a commit
that referenced
this pull request
May 23, 2026
The README's "COBOL Code Analysis" feature row was stale relative to what PR #50 / #51 / #52 actually shipped — no mention of dynamic CALL resolution, DB2 column-level pairing, REPLACING-aware inferred matching (Phase C), or the eval harness. Replace with a "Language Plugins" framing: - Renamed feature row to "Language Plugins" with a one-line summary that points at the new section. - New "Language Plugins" section after Integration: COBOL shipped (current capabilities), JCL planned (PRD Phase 2), plus the eval harness + NIST baseline as the precision yardstick for future plugin work. Doc-only, no code change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three small lever follow-ups to the eval-harness work (#50 / #51), each catching a class of regression the prior PRs left open:
f341203— FILLER filter: COBOL's name-for-unnamed never carries cross-copybook identity evidence. NIST CCVS dogfood surfaced one such pair; filter at the name-keyed candidate-sourcing layer drops them. Semantic tier already excludes FILLER via independent paths (PIC+USAGE filter + same-name leaf gate).892773d— callBound confidence-tier pinning: same shape as PR feat(cobol): Phase C — REPLACING-aware inferred field lineage #51's deterministic linkage pin. Closes the parallel precision gap on CALL-USING — a builder regression that emits the right pair under the wrong confidence tier (deterministicshape-match landing ashigh, losing pictureMatch evidence) is now caught.e9fc21f— NIST CCVS dogfood baseline: per-fixture eval tests catch "did manifest entries emit"; baseline catches "did builder start emitting something the manifests don't pin." Committed slice of NIST COBOL85 (SM* programs + K* copybooks, ~780 KB, public-domain) + a byte-exact baseline regression test..gitattributeslocks LF so Windows clones withautocrlf=truedon't break the comparison.Why
The eval-harness foundation from #50 / #51 is the precision yardstick. These PRs use it for what it was built for — surface concrete regressions and trade-offs that were previously invisible.
Test plan
npm test— 3950 / 3950 pass (1 FILLER unit test + 4 callBound tier tests + 2 dogfood-baseline tests are new)npm run build— TypeScript clean[].gitattributesscope is right (currently locks LF on .txt/.md/.ts/.js/.json/.yaml/.yml/.cbl/.cob/.cpy)🤖 Generated with Claude Code