Skip to content

Commit 2f206cc

Browse files
ychanclaude
andcommitted
test(auto-eval): pin EDGAR_DETERMINISM_DEGRADED parsing contract
Address review comment #1 (Important) on PR #764 Sub-project A. The ``get_decision_threshold()`` helper landed in Sub-project A without direct test coverage — the spec explicitly called for "the helper + its tests." Sub-project B's chokepoint will consume this helper, so the env-var parsing contract needs to be pinned before that wiring lands, not after. 12 cases across 4 classes of behavior: - Unset env var → normal (0.005) - Exactly "1" → degraded (0.01) - 9 rejection cases: "0", "true", "TRUE", "yes", "", " 1", "1 ", "01", "2" (strict ``== "1"`` parsing — no stripping, no coercion, no bool-spelling) - Invariant: degraded is always wider than normal Uses monkeypatch to avoid env pollution across tests. If a future change loosens the parsing, these tests will fail loudly and force a contract update in a single place instead of inside the chokepoint's fast path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 7f53663 commit 2f206cc

File tree

1 file changed

+84
-0
lines changed

1 file changed

+84
-0
lines changed
Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
"""
2+
Unit tests for ``get_decision_threshold()`` — the chokepoint decision threshold
3+
helper introduced in PR #764 Sub-project A.
4+
5+
The helper is **unwired** in Sub-project A; Sub-project B's chokepoint will
6+
consume it when it lands. These tests pin the env-var parsing contract so that
7+
future wiring can trust a single, documented semantics without surprises at
8+
merge time.
9+
10+
Contract under test:
11+
- Unset env var → normal threshold (0.005)
12+
- Exactly ``"1"`` → degraded threshold (0.01)
13+
- Any other value → normal threshold (strict parsing)
14+
- Degraded is always wider → invariant
15+
16+
The parsing is intentionally strict (``os.environ.get(...) == "1"``). Any
17+
change to this contract (e.g., accepting ``"true"`` / ``"yes"``) should update
18+
this test file so the break is caught at the test layer, not inside the
19+
chokepoint's fast path.
20+
"""
21+
22+
import pytest
23+
24+
from edgar.xbrl.standardization.tools.auto_eval import (
25+
_DECISION_THRESHOLD_DEGRADED,
26+
_DECISION_THRESHOLD_NORMAL,
27+
get_decision_threshold,
28+
)
29+
30+
31+
class TestDecisionThreshold:
32+
"""Pin the EDGAR_DETERMINISM_DEGRADED env-var parsing contract."""
33+
34+
def test_unset_env_var_returns_normal_threshold(self, monkeypatch):
35+
"""Default path: no env var set → normal (narrow) threshold."""
36+
monkeypatch.delenv("EDGAR_DETERMINISM_DEGRADED", raising=False)
37+
assert get_decision_threshold() == _DECISION_THRESHOLD_NORMAL
38+
assert get_decision_threshold() == pytest.approx(0.005)
39+
40+
def test_exactly_one_returns_degraded_threshold(self, monkeypatch):
41+
"""Documented escape hatch: EDGAR_DETERMINISM_DEGRADED=1 → degraded."""
42+
monkeypatch.setenv("EDGAR_DETERMINISM_DEGRADED", "1")
43+
assert get_decision_threshold() == _DECISION_THRESHOLD_DEGRADED
44+
assert get_decision_threshold() == pytest.approx(0.01)
45+
46+
@pytest.mark.parametrize(
47+
"value",
48+
[
49+
"0", # false-ish int — strict parser must reject
50+
"true", # common bool spelling — strict parser must reject
51+
"TRUE", # uppercase variant
52+
"yes", # another common bool spelling
53+
"", # empty string — strict parser must reject
54+
" 1", # leading space — strict parser must reject (no stripping)
55+
"1 ", # trailing space — strict parser must reject
56+
"01", # numeric padding — strict parser must reject
57+
"2", # other int — strict parser must reject
58+
],
59+
)
60+
def test_any_value_other_than_exact_one_falls_through_to_normal(
61+
self, monkeypatch, value
62+
):
63+
"""Strict ``== "1"`` parsing: anything else returns the normal threshold.
64+
65+
If this test fails, the helper's parsing contract has drifted.
66+
Document the new contract here AND in
67+
``edgar/xbrl/standardization/tools/auto_eval.py`` so Sub-project B's
68+
chokepoint can rely on a single source of truth.
69+
"""
70+
monkeypatch.setenv("EDGAR_DETERMINISM_DEGRADED", value)
71+
assert get_decision_threshold() == _DECISION_THRESHOLD_NORMAL, (
72+
f"EDGAR_DETERMINISM_DEGRADED={value!r} unexpectedly triggered degraded "
73+
f"mode. Parsing is strict '== \"1\"' by design."
74+
)
75+
76+
def test_degraded_threshold_is_wider_than_normal(self):
77+
"""Invariant: degraded mode must widen the gate, never narrow it.
78+
79+
Degraded mode is the escape hatch for measurement noise — widening the
80+
acceptance window is the whole point. If this ever flips, the chokepoint
81+
would become *more* strict under degraded conditions, which is the
82+
opposite of the intended behavior.
83+
"""
84+
assert _DECISION_THRESHOLD_DEGRADED > _DECISION_THRESHOLD_NORMAL

0 commit comments

Comments
 (0)