map: one versioned capability contract instead of five copies - #137
Merged
Conversation
|
Cohesively centralizes and version-controls the capability contract with robust codegen and tests. 🎯 Quality: 94% Elite · 📦 Size: Large — consider splitting if possible 📈 This month: Your 57th PR — above team average · Averaging Excellent |
patchstackdave
force-pushed
the
davejong/map-capability-contract
branch
from
August 17, 2026 09:11
cc7f8cc to
f755047
Compare
Contributor
Author
|
/review |
daniloradovic
approved these changes
Aug 17, 2026
The closed vocabularies that describe what the map can see — sink kinds, argument roles, candidate families, confidence tiers — were declared independently in five places: the TypeScript unions here, and separate copies in the rule-authoring and rule-binding layers that consume the map. Adding a capability meant editing all of them, and the failure mode of missing one is silent: a sink kind this side emits and an authoring layer rejects makes the capability unauthorable, one the binding layer does not know makes every flow of that kind unusable. Nothing errors. The capability simply never matches, which reads exactly like "not reachable". That is the same shape as the wrong-pin bugs in the map hardening track: three instances, one lossy representation. The fix is the representation. - `src/map/capabilities.ts` is now the single definition, versioned, with the reasoning for why a new member is a contract change and not a list edit. - The TypeScript unions are DERIVED from it (`(typeof SINK_KINDS)[number]`) rather than declared beside it, which would have been a second copy with the same drift risk. - `capabilities.json` is generated and committed, so a vocabulary change is a reviewable diff in the contract. It is not added to package.json `files`: the consumers vendor from source, and the published surface is separately reviewed. - Tests defend the contract, not just the values: the committed manifest must be byte-identical to what the emitter produces (verified by adding a sink kind and watching it fail), the manifest must agree with the TS member for member, the auto-promotable tier must remain the single strongest proven tier, and everything the extractor actually emits must fall inside the vocabulary — with a non-vacuity check first, since a fixture producing no sinks would satisfy that trivially.
…e must bump the version Two ways the contract could still drift silently. A capability could be declared with nothing behind it. Adding a sink kind, regenerating the manifest and teaching every consumer to accept it succeeded even if no recognizer ever emitted that kind — a vocabulary entry with no behaviour, unlocking a rule family that can never fire. The existing "everything emitted is inside the vocabulary" test cannot see it: that only checks the kinds that DO emit are legal, and a fixture exercising the other four still passes a "we emit some sinks" assertion. Each kind now names its own control — the minimal shape that must produce a sink of exactly that kind — and the tests run every control, assert the kind appears, and for a rule-generatable kind assert a candidate with a real family compiles. Because the matrix is typed Record<SinkKind, …>, adding a kind without a control is a TYPE error: the compiler asks before CI does. A kind that can compile a rule can also compile a WRONG one, so those additionally require an adversarial lookalike in the corpus, asserted rather than assumed. And the version could stay put. A member added or removed without moving CAPABILITY_VERSION left every vendoring consumer unable to tell it was behind — the same silent drift the manifest exists to remove, one level up. check-capability-version.mjs compares the manifest against the base branch and classifies: a member or field added needs a minor bump, a member removed or a scalar redefined is BREAKING and needs a major one, because a consumer pinned to the old list keeps emitting a value that can no longer match. Wired as its own CI job with full history. Both verified by simulation: declaring an unrecognized capability fails three tests (no sink, no candidate, no adversarial coverage), and the four version paths — additive without a bump, additive with one, a removal under a minor bump, and a redefined scalar — each behave as specified.
…s is exact Two ways the coverage checks could pass without proving anything. Adversarial coverage was a text search over the corpus file: any occurrence of an API name counted, so a comment, a positive fixture, or an unrelated case satisfied it. That is coverage that can be true by coincidence — the same substring reasoning this suite exists to reject. The cases now live in tests/map/corpus-cases.ts with stable ids, and each rule-generatable capability names the adversarial case that covers it. The test resolves that id to a real case, asserts it is in the adversarial category, and asserts it declares no expected candidates — then BUILDS it and checks the extractor agrees, including the corpus non-vacuity rule, since a case that detects nothing at all would satisfy a zero-candidate assertion for the wrong reason. That exposed a genuine gap while wiring it up: eval had no adversarial case. It is recognized only as a bare global call or `new Function`, gated on the name not being locally bound, and nothing asserted the gate. adv/local-eval-lookalikes covers it — a locally-declared `Function` and a member `.eval()` on an untraceable receiver, neither of which may compile code injection. And the controls only required "some rule-generatable flow with some family". A recognizer that classified a flow as the wrong mitigation class would have passed, which is a rule that inspects the wrong thing and blocks the wrong traffic. Each rule-generatable control now declares its exact argumentRole and candidateFamily and the test asserts equality. The type is a discriminated union, so a capability declared rule-generatable without a role, a family and a case id does not compile. Verified by simulation: a wrong family fails, an id that resolves to nothing fails, and an id pointing at a case that legitimately expects candidates fails.
patchstackdave
force-pushed
the
davejong/map-capability-contract
branch
from
August 17, 2026 10:25
016cac7 to
0727b66
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The closed vocabularies that describe what the map can see — sink kinds, argument roles, candidate families, confidence tiers — were declared independently in five places: the TypeScript unions here, plus separate copies in the layers that consume the map (rule authoring, and the platform that binds a coordinate into a rule).
Adding a capability meant editing all of them, and missing one fails silently: a sink kind this side emits but an authoring layer rejects makes the capability unauthorable; one the binding layer doesn't know makes every flow of that kind unusable. Nothing throws — the capability just never matches, which reads exactly like "not reachable".
That is the same shape as the wrong-pin bugs in the map hardening track: three instances, one lossy representation. The fix is the representation, not a third patch.
What
src/map/capabilities.tsis the single definition, versioned, with the reasoning that a new member is a contract change rather than a list edit — and that a new sink family owes adversarial corpus cases before it may generate rules, since one capability admits a whole package family at once.(typeof SINK_KINDS)[number]) rather than declared beside it, which would have been a second copy with the same drift risk.capabilities.jsonis generated and committed, so a vocabulary change shows up as a reviewable diff in the contract instead of appearing silently indist.npm run capabilitiesregenerates;--checkverifies.package.jsonfiles. The consumers vendor it from source, and the published surface is separately reviewed — the existing pack-safety test caught my first attempt at widening it, which was the correct outcome.Verification
977 tests, typecheck clean. The drift guard was verified by actually causing drift: adding a member to
SINK_KINDSwithout regenerating fails with "capabilities.json is stale — runnpm run capabilities".The tests defend the contract rather than the values: the manifest must be byte-identical to the emitter's output, must agree with the TS member for member, the auto-promotable tier must remain the single strongest proven tier, and everything the extractor actually emits must fall inside the vocabulary — with a non-vacuity check first, since a fixture producing no sinks would satisfy that trivially.
Consuming layers
The vocabulary is consumed outside this repo. Those layers now vendor
capabilities.jsonand assert conformance against it, so a stale copy is a test failure rather than a silent mismatch. One consumer is still a hand-maintained copy and is tracked separately — until it reads the manifest, it remains the one place drift can occur.