Skip to content

Assert the comparison that decides whether an entry moved [#44] - #204

Merged
iderex merged 1 commit into
mainfrom
tests/the-comparison-that-keeps-the-record-honest-is-asserted-field-by-field
Aug 26, 2026
Merged

iderex merged 1 commit into
mainfrom
tests/the-comparison-that-keeps-the-record-honest-is-asserted-field-by-field

Conversation

@iderex

@iderex iderex commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Under #44. It does not close it: what that issue still waits on is where a
mutation run lives, which is a call rather than something a reading settles,
and this change adds no leg to the gate.

What was wrong

The refresh verb reports every repository as NEW, MOVED, GONE or unchanged,
and what decides MOVED is same, comparing four fields. Its own comment says
it exists to keep that comparison honest. Nothing asserted it.

The only case that reached it moved two fields at once, so a comparison that
had stopped reading one field still answered MOVED there and the suite stayed
green. What that costs is the line somebody reads instead of diffing the
record: a repository whose count moved, announced as unchanged.

How it was found

By re-taking the mutation run #44 records, rather than by reading the code.
The tool, the coefficient and the clone outside the working tree are the ones
that issue already argues for, and the binary still reports dev rather than
the tag it was built from:

gremlins --version
gremlins version dev windows/amd64

At 440c545, which is where the score on #44 was last taken:

git checkout 440c545
gremlins unleash --timeout-coefficient=20 --workers=2 ./internal/releases
Killed: 20, Lived: 0, Not covered: 14
Test efficacy: 100.00%
Mutator coverage: 58.82%

At ad8b25c, the mainline this branch is off:

git checkout ad8b25c
gremlins unleash --timeout-coefficient=20 --workers=2 ./internal/releases
Killed: 37, Lived: 5, Not covered: 30
Test efficacy: 88.10%
Mutator coverage: 58.33%

One commit touches the package between those two:

git log --oneline 440c545..ad8b25c -- internal/releases/
b3768b5 Take the server generation from what each release published [#91]

The five, and four of them are the four field comparisons:

gremlins unleash --timeout-coefficient=20 --workers=2 ./internal/releases | grep LIVED
       LIVED CONDITIONALS_BOUNDARY at releases.go:139:17
       LIVED CONDITIONALS_NEGATION at releases.go:286:16
       LIVED CONDITIONALS_NEGATION at releases.go:286:47
       LIVED CONDITIONALS_NEGATION at releases.go:286:78
       LIVED CONDITIONALS_NEGATION at releases.go:286:114

All runs 2026-08-26, in a clone outside the working tree because the tool
rewrites source files where it finds them.

What the change does

One case, comparing an entry against itself and against five neighbours that
differ in exactly one thing.

Negating any of the four comparisons turns the inequality into an equality,
which makes an entry differ from itself, so the self comparison reds on all
four; the neighbour names which field went unread. Each mutation applied by
hand at the commit being pushed, one at a time, with the tree restored
between:

a.Finished != b.Finished  ->  a.Finished == b.Finished
--- FAIL: TestOneFieldIsEnoughToMakeTwoEntriesDifferent
    releases_test.go:250: an entry was reported as saying something other than itself
    releases_test.go:260: one more finished release: two entries that differ were reported as saying the same thing

a.Prereleases != b.Prereleases  ->  a.Prereleases == b.Prereleases
--- FAIL: TestOneFieldIsEnoughToMakeTwoEntriesDifferent
    releases_test.go:250: an entry was reported as saying something other than itself
    releases_test.go:260: one more prerelease: two entries that differ were reported as saying the same thing

a.Unstated != b.Unstated  ->  a.Unstated == b.Unstated
--- FAIL: TestOneFieldIsEnoughToMakeTwoEntriesDifferent
    releases_test.go:250: an entry was reported as saying something other than itself
    releases_test.go:260: one more stating no server generation: two entries that differ were reported as saying the same thing

len(a.Generations) != len(b.Generations)  ->  len(a.Generations) == len(b.Generations)
--- FAIL: TestOneFieldIsEnoughToMakeTwoEntriesDifferent
    releases_test.go:250: an entry was reported as saying something other than itself
panic: runtime error: index out of range [1] with length 1

The fourth is worth reading rather than skipping. The length term is not a
shortcut: it bounds the loop under it, so negating it walks past the end of
the shorter slice. It reds by assertion first and by that panic second.

The score with the case in, in the same clone at the same commit:

gremlins unleash --timeout-coefficient=20 --workers=2 ./internal/releases
Killed: 42, Lived: 1, Not covered: 29
Test efficacy: 97.67%
Mutator coverage: 59.72%

Run 2026-08-26. One mutant also moves out of Not covered, because nothing
reached same directly before.

The survivor that is left, and why it is left

CONDITIONALS_BOUNDARY at releases.go:139:17 is the boundary on the count of
finished releases in the refusal about a finished release with no server
generation. Three guards stand above it:

git show origin/main:internal/releases/releases.go | sed -n '128,139p'
    if r.Finished < 0 || r.Prereleases < 0 {
...
    if r.Unstated < 0 {
...
    if r.Unstated > r.Finished {
...
    if r.Finished > 0 && len(r.Generations) == 0 && r.Unstated < r.Finished {

So an entry reaching that line has a count that is not negative, a count of
releases stating no generation that is not negative, and no more of the
second than of the first. The only inputs the boundary changes the verdict
for are those with no finished release, and there the other two guards force
the second count to zero as well, so the last term reads zero against zero
either way. That is a reading of those four lines rather than a measurement:
the suite staying green under the mutation is consistent with it and proves
nothing on its own.

go test -count=1 ./...    with the mutation applied
ok  github.com/Flowfin/site/internal/releases 0.412s
21 package(s), every one ok

Chasing it would mean changing the refusal to suit the tool, which is the
wrong direction for a package whose guards are the point.

The gate at the commit being pushed

go run . ci
gate: 7 legs, in order: format, vet, test, build, links, sitemap, invariants
  needs-network was not asked for. Asking costs a request to the public name from whatever machine runs it, and a verdict that moves when somebody else's service does rather than when this tree changes. Ask with: go run ./harness/needs-network
  format: ok, 74 file(s)
  vet: ok
  test: ok, 37 test file(s)
  build: ok, 22 file(s)
  links: every reference that stays inside this site resolves to a file the build wrote
  sitemap: every page the build wrote is listed once, and every entry has a page behind it
  invariants: ok, 39 rule(s) decided, 1 owed and not decided
7 of 7 legs ran. None was skipped.

go run . hygiene origin/main HEAD
hygiene: 1 non-merge commit(s) in origin/main..HEAD, origin internal
  f9c55b905ce5: subject carries its reference
1 commit(s) judged, none refused.

git diff --name-only origin/main...HEAD
internal/releases/releases_test.go

One file, one topic, no production code touched.

No second reader

This change carries no second reader. The evidence in place of one is the
four mutations above, each applied at this commit and each producing a red
run naming the field it removed, plus the before and after scores from the
same tool in the same clone.

The refresh reports each repository as NEW, MOVED, GONE or unchanged, and
what decides MOVED is one function comparing four fields. Nothing asserted
that function. The only case reaching it moves two fields at once, so a
comparison that had stopped reading one of them still answered MOVED there
and the suite stayed green. The failure that prevents is a change to a
record being announced as unchanged, which is the one line somebody reads
instead of diffing the file.

Found by re-taking the mutation run this issue records. Between 440c545 and
ad8b25c the package's efficacy fell from 100.00% to 88.10% with one commit
touching it, and four of the five survivors were the four field
comparisons: negating any one of them left every test passing.

The new case compares an entry against itself and against five neighbours
that differ in exactly one thing. Negating a comparison makes an entry
differ from itself, so the self case reds on all four; the neighbours name
which field went unread. The length term also bounds the loop below it, and
negating it indexes past the shorter slice.

The fifth survivor is a boundary on `r.Finished > 0` at the same refusal.
The three guards above it leave every entry with a count that is not
negative and no more releases stating no generation than were published, so
the only inputs the boundary changes the verdict for are refused two guards
earlier and no record distinguishes the two spellings. It is left alive
rather than chased.

Signed-off-by: Nils Lehnen <30603423+iderex@users.noreply.github.com>
@iderex
iderex merged commit 9a2bf51 into main Aug 26, 2026
17 checks passed
@iderex
iderex deleted the tests/the-comparison-that-keeps-the-record-honest-is-asserted-field-by-field branch August 26, 2026 23:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant