test(classifier): kill 30 of 32 surviving mutants in classify_task - #80
Merged
Merged
Conversation
The gate reported 32 surviving mutants in this file, every one inside
classify_task - despite 22 existing tests. The reason is visible in those
tests: they assert r.complexity and nothing else. complexity is a four-way
bucket, so flipping "score += 2" to "score -= 2" usually lands in the same
bucket and the test still passes.
The lever these tests use instead is CONFIDENCE, which is
|score| / (signals.len() * 4). It is a precise function of the score, so
asserting it pins the arithmetic itself. Paired with an exact signals vector -
which carries the measured counts, e.g. "word_count>50 (51)" - every branch
and every += becomes observable. A shared assert_classified helper checks all
three together, because complexity alone is too coarse to see an arithmetic
change and confidence alone cannot say which branch fired.
Method: expected values were derived from an independent model of the
algorithm, and that model was validated against the real implementation on 10
probe inputs BEFORE any assertion was written. None of these numbers were read
off the code.
Boundary inputs are the other half. Mutants at a comparison only die when the
input sits exactly on it: 19/20 words, exactly 50, exactly 200, exactly 20
prior messages. Several also need a second signal present, because the sign of
a += is invisible when it is the only term - "51 words + one complex keyword"
is what kills "+= 2 -> -=", since the minus version lands in Simple at 0.25
instead of Complex at 0.75.
classifier.rs drops 487 -> 240 lines; the 22 pre-existing tests move verbatim
into classifier/tests.rs alongside 21 new ones.
Verified by applying 19 mutants: 17 KILLED.
The two survivors are EQUIVALENT and are documented in the file rather than
chased. Both were predicted before running them, and the prediction verified:
len() > 100_000 -> >= : at exactly 100_000 bytes the mutant slices
[..100_000], which is the whole string. Identical result.
confidence < 0.3 -> <= : the fallback needs confidence exactly 0.3 AND
complexity Simple. Confidence is |score| / (signals * 4) with |score| <= 2
for Simple, so 0.3 would require signals = 1.67. Unreachable.
So this file's ceiling is 30 of 32, not 32 of 32.
cargo test --lib intelligence::classifier passes (43); clippy --all-targets
--all-features -D warnings and cargo fmt --check both exit 0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gate reported 32 surviving mutants in this file, every one inside
classify_task— despite 22 existing tests.The reason is visible in those tests: they assert
r.complexityand nothing else.complexityis a four-way bucket, so flippingscore += 2toscore -= 2usually lands in the same bucket and the test still passes.The lever: confidence reveals the score
confidenceis|score| / (signals.len() * 4)— a precise function of the score. Asserting it pins the arithmetic itself. Paired with an exactsignalsvector, which carries the measured counts ("word_count>50 (51)"), every branch and every+=becomes observable.A shared
assert_classifiedhelper checks all three together:complexityalone is too coarse to see an arithmetic change, andconfidencealone cannot say which branch fired.Method
Expected values were derived from an independent model of the algorithm, and that model was validated against the real implementation on 10 probe inputs before any assertion was written. None of these numbers were read off the code — that would produce tests asserting whatever the code happens to do.
Boundary inputs are the other half
Mutants at a comparison only die when the input sits exactly on it: 19/20 words, exactly 50, exactly 200, exactly 20 prior messages.
Several also need a second signal present, because the sign of a
+=is invisible when it is the only term. "51 words + one complex keyword" is what kills+= 2→-=: the minus version lands in Simple at confidence 0.25 instead of Complex at 0.75.classifier.rsdrops 487 → 240 lines; the 22 pre-existing tests move verbatim alongside 21 new ones.Verification — 17 of 19 killed
wc < 20→<=wc > 200→>=wc > 50→>=/<score += 4→*=score += 2→-=(word count)/ 2→% 2/* 2complex_kw > 0→>= 0count * 4→count + 4message_count > 20→>=score += 2→-=(history)3..=5/6..=9total_signals * 4.0→+ 4.0abs / max→abs % maxlen() > 100_000→== 100_000len() > 100_000→>=confidence < 0.3→<=The two survivors are equivalent, and were predicted
Both were called before running them, then the prediction was verified:
len() > 100_000→>=: at exactly 100,000 bytes the mutant slices[..100_000], which is the whole string. Identical result.confidence < 0.3→<=: the fallback needs confidence exactly0.3and complexitySimple. Confidence is|score| / (signals * 4)with|score| <= 2for Simple, so0.3would requiresignals = 1.67. Unreachable.So this file's ceiling is 30 of 32, not 32 of 32. Documented in the file so the eventual gate number is read correctly rather than chased.
cargo test --lib intelligence::classifierpasses (43);clippy --all-targets --all-features -- -D warningsandcargo fmt --checkboth exit 0.