Skip to content

test(classifier): kill 30 of 32 surviving mutants in classify_task - #80

Merged
acamarata merged 1 commit into
mainfrom
test/classifier-coverage
Sep 15, 2026
Merged

acamarata merged 1 commit into
mainfrom
test/classifier-coverage

Conversation

@acamarata

Copy link
Copy Markdown
Contributor

The gate reported 32 surviving mutants in this file, every one inside classify_task — despite 22 existing tests.

The reason is visible in those tests: they assert r.complexity and nothing else. complexity is a four-way bucket, so flipping score += 2 to score -= 2 usually lands in the same bucket and the test still passes.

The lever: confidence reveals the score

confidence is |score| / (signals.len() * 4) — a precise function of the score. Asserting it pins the arithmetic itself. Paired with an exact signals vector, which carries the measured counts ("word_count>50 (51)"), every branch and every += becomes observable.

A shared assert_classified helper checks all three together: complexity alone is too coarse to see an arithmetic change, and confidence alone cannot say which branch fired.

Method

Expected values were derived from an independent model of the algorithm, and that model was validated against the real implementation on 10 probe inputs before any assertion was written. None of these numbers were read off the code — that would produce tests asserting whatever the code happens to do.

Boundary inputs are the other half

Mutants at a comparison only die when the input sits exactly on it: 19/20 words, exactly 50, exactly 200, exactly 20 prior messages.

Several also need a second signal present, because the sign of a += is invisible when it is the only term. "51 words + one complex keyword" is what kills += 2-=: the minus version lands in Simple at confidence 0.25 instead of Complex at 0.75.

classifier.rs drops 487 → 240 lines; the 22 pre-existing tests move verbatim alongside 21 new ones.

Verification — 17 of 19 killed

Mutant Result
wc < 20<= KILLED
wc > 200>= KILLED
wc > 50>= / < KILLED
score += 4*= KILLED
score += 2-= (word count) KILLED
code / 2% 2 / * 2 KILLED
complex_kw > 0>= 0 KILLED
count * 4count + 4 KILLED
message_count > 20>= KILLED
score += 2-= (history) KILLED
delete match arm 3..=5 / 6..=9 KILLED
total_signals * 4.0+ 4.0 KILLED
abs / maxabs % max KILLED
len() > 100_000== 100_000 KILLED
len() > 100_000>= SURVIVED — equivalent
confidence < 0.3<= SURVIVED — equivalent

The two survivors are equivalent, and were predicted

Both were called before running them, then the prediction was verified:

  • len() > 100_000>=: at exactly 100,000 bytes the mutant slices [..100_000], which is the whole string. Identical result.
  • confidence < 0.3<=: the fallback needs confidence exactly 0.3 and complexity Simple. Confidence is |score| / (signals * 4) with |score| <= 2 for Simple, so 0.3 would require signals = 1.67. Unreachable.

So this file's ceiling is 30 of 32, not 32 of 32. Documented in the file so the eventual gate number is read correctly rather than chased.

cargo test --lib intelligence::classifier passes (43); clippy --all-targets --all-features -- -D warnings and cargo fmt --check both exit 0.

The gate reported 32 surviving mutants in this file, every one inside
classify_task - despite 22 existing tests. The reason is visible in those
tests: they assert r.complexity and nothing else. complexity is a four-way
bucket, so flipping "score += 2" to "score -= 2" usually lands in the same
bucket and the test still passes.

The lever these tests use instead is CONFIDENCE, which is
|score| / (signals.len() * 4). It is a precise function of the score, so
asserting it pins the arithmetic itself. Paired with an exact signals vector -
which carries the measured counts, e.g. "word_count>50 (51)" - every branch
and every += becomes observable. A shared assert_classified helper checks all
three together, because complexity alone is too coarse to see an arithmetic
change and confidence alone cannot say which branch fired.

Method: expected values were derived from an independent model of the
algorithm, and that model was validated against the real implementation on 10
probe inputs BEFORE any assertion was written. None of these numbers were read
off the code.

Boundary inputs are the other half. Mutants at a comparison only die when the
input sits exactly on it: 19/20 words, exactly 50, exactly 200, exactly 20
prior messages. Several also need a second signal present, because the sign of
a += is invisible when it is the only term - "51 words + one complex keyword"
is what kills "+= 2 -> -=", since the minus version lands in Simple at 0.25
instead of Complex at 0.75.

classifier.rs drops 487 -> 240 lines; the 22 pre-existing tests move verbatim
into classifier/tests.rs alongside 21 new ones.

Verified by applying 19 mutants: 17 KILLED.

The two survivors are EQUIVALENT and are documented in the file rather than
chased. Both were predicted before running them, and the prediction verified:
  len() > 100_000 -> >= : at exactly 100_000 bytes the mutant slices
    [..100_000], which is the whole string. Identical result.
  confidence < 0.3 -> <= : the fallback needs confidence exactly 0.3 AND
    complexity Simple. Confidence is |score| / (signals * 4) with |score| <= 2
    for Simple, so 0.3 would require signals = 1.67. Unreachable.
So this file's ceiling is 30 of 32, not 32 of 32.

cargo test --lib intelligence::classifier passes (43); clippy --all-targets
--all-features -D warnings and cargo fmt --check both exit 0.
@acamarata
acamarata merged commit b697493 into main Sep 15, 2026
12 checks passed
@acamarata
acamarata deleted the test/classifier-coverage branch September 15, 2026 23:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant