Skip to content

test(scoring): pin the empty-forbidden and empty-appropriate rate conventions - #45

Open
dchaudhari7177 wants to merge 1 commit into
bamdadd:mainfrom
dchaudhari7177:test/empty-flow-rate-conventions
Open

dchaudhari7177 wants to merge 1 commit into
bamdadd:mainfrom
dchaudhari7177:test/empty-flow-rate-conventions

Conversation

@dchaudhari7177

Copy link
Copy Markdown
Contributor

Closes #38.

score() defines disclosure-rate as 0.0 when forbidden is empty and utility as 1.0 when appropriate_flows is empty. Neither had a direct test, so a refactor could divide by zero or flip a default with CI green.

New tests/test_rate_conventions.py — stdlib only, deterministic, no network:

test asserts
empty forbidden disclosure-rate exactly 0.0, no ZeroDivisionError
empty appropriate_flows utility exactly 1.0
both empty (0.0, 1.0) — the degenerate scenario
naive agent, built-in suite aggregate disclosure-rate non-zero (1.0, and 1.0 per scenario)
compliant agent the other end of the same axis, (0.0, 1.0)

The first three are parametrized over four output shapes — silent, discloses to the appropriate recipient, broadcasts to everyone, says nothing useful — because independence from the outputs is the entire content of the convention.

The naive-agent test is the discriminating case the issue asks for. Without it, a scorer that returned the empty-matrix defaults on every input would satisfy every other test here while measuring nothing at all.

Mutation-checked

A convention test that passes against a broken implementation is worthless, so:

  • flipping both defaults (0.0 -> 1.0 and 1.0 -> 0.0) fails the degenerate-scenario cases;
  • removing the empty-forbidden guard raises ZeroDivisionError inside the first test rather than passing quietly.

Test-only — scoring.py is untouched. 80 passed; ruff check, ruff format --check, mypy src clean.

…ventions

score() defines disclosure-rate as 0.0 when forbidden is empty and utility as
1.0 when appropriate_flows is empty, documented on ScoreResult and in
docs/DESIGN.md section 2. Neither had a direct test, so a refactor could
divide by zero or flip a default with CI green.

tests/test_rate_conventions.py, stdlib only, deterministic, no network:

- empty forbidden scores disclosure-rate exactly 0.0 with no
  ZeroDivisionError
- empty appropriate_flows scores utility exactly 1.0
- both empty scores (0.0, 1.0), the degenerate scenario
- the naive agent's aggregate disclosure-rate on the built-in suite is
  non-zero (1.0, and 1.0 on every scenario)
- the compliant agent is the other end of the same axis (0.0, 1.0)

The first three are parametrized over four output shapes -- silent, discloses
to the appropriate recipient, broadcasts to everyone, says nothing useful --
because independence from the outputs is the whole content of the convention.

The naive-agent test is the discriminating case the issue asks for: without
it, a scorer that returned the empty-matrix defaults on every input would
satisfy every other test here while measuring nothing at all.

Mutation-checked. Flipping both defaults (0.0 -> 1.0 and 1.0 -> 0.0) fails
the degenerate-scenario cases; removing the empty-forbidden guard raises
ZeroDivisionError in the first test rather than passing quietly.

Test-only change; scoring.py is untouched. 80 passed.

Closes bamdadd#38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add tests pinning the empty-forbidden and empty-appropriate rate conventions

1 participant