fix(verify): close GNN allele-blindness, missing package data, and a silent mock-fallback report gap - #296
Conversation
…silent mock-fallback report gap
StructuralPeptideMHCDataset loaded its MHC pseudo-sequence table via a
CWD-relative path. Any CWD but the repo root - including an installed
sestrav package invoked from anywhere else - made every allele fall
through to an identical all-alanine placeholder with no error, only a
per-allele warning. generate_canonical_groove_coords never uses its
allele_name argument either, so this table was the only source of
allele signal anywhere in the graph.
Scope: src/verify/structural_gnn.py is the SESTRAV-VERIFY evaluation
harness and the only production importer of this class. The v2.3
production GNN path (src/gnn/, GraphPredictorV2) has zero allele
references and is unaffected; GNN promotion remains deferred and no
certified result passes through this code path.
Three fixes, landed together because none alone closes the gap:
- Resolve the table via Path(__file__).resolve().parent instead of a
CWD-relative path; raise on an absent OR content-empty table rather
than silently defaulting to {}.
- Declare [tool.setuptools.package-data] so a built wheel actually
ships the JSON files - verified separately: the wheel carried zero
non-.py members before this change.
- sestrav_evaluator.py's except arms already caught a construction
failure and fell back to mock scores, but did so silently, leaving
run_evaluation_pipeline's top-level use_mock_fallback flag False
even when a virus's scores were actually mock. Threaded a
used_mock_fallback flag through both except arms into each virus's
result, and recompute the top-level flag as an OR across every
virus after the per-virus loop.
Twelve new/rewritten tests. Every guard and every fallback-flag site
was individually mutation-tested by reverting it in isolation and
confirming its dedicated test fails, then restoring it.
Signed-off-by: Gavin Borges <gavinmborges1104@gmail.com>
…pseudo-sequence-loading # Conflicts: # CHANGELOG.md
Post-review corrections (pushed as
|
| File | main | branch | added | removed |
|---|---|---|---|---|
test_structural_gnn.py |
11 | 14 | 4 | 1 |
test_sestrav_evaluator_gnn.py |
5 | 8 | 3 | 0 |
Seven added, three rewritten, one removed - ten, not twelve. Corrected, and the removed test is now named explicitly (test_dataset_no_pseudo_seqs_file, which pinned the silent-empty-table behaviour this PR deliberately converts into a raise) rather than quietly dropped. A reviewer finds this with one grep -c.
Also confirmed by the audit, unchanged
Every empirical claim in the original PR body re-verified independently: the wheel from main has exactly 75 entries with 6 non-.py members all inside .dist-info/ and no JSON; the branch wheel has 77 with both; and "the two tracked non-.py files under the packaged source trees" is exactly right (git ls-tree over src/, sestrav/, functions/ returns precisely those two).
Merged with main after #294 and #295 landed; conflict was in CHANGELOG.md only and is resolved as described above. Branch scope re-verified as 6 files, no merge leakage. 38 tests pass, ruff clean, integrity harness at 151 PASS / 0 WARN / 2 FAIL / 7 SKIP.
StructuralPeptideMHCDataset.__init__(src/verify/structural_gnn.py) loaded its 34-residue MHC pocket-sequence table via a CWD-relativePath("src/verify/mhc_pseudo_sequences.json"). Any process CWD but the repo root - including an installedsestravpackage invoked from anywhere else - made that path silently fail to resolve,pseudo_seqsdefault to{}, and every HLA allele fall through to an identical"A"*34all-alanine placeholder.generate_canonical_groove_coordsnever uses its ownallele_nameargument either, so this table was the only source of allele signal anywhere in the graph: two peptide-allele pairs sharing a peptide would produce byte-identical MHC pocket node features and edge attributes regardless of true allele identity - silently, with only a per-allelelogger.warning.Scope - please read before the diff
src/verify/structural_gnn.pyis the SESTRAV-VERIFY evaluation harness (sestrav_evaluator.py's report, gatingpromote_gnn.py's promotion scorecard). It is the only production importer ofStructuralPeptideMHCDatasetin the repository (grep-confirmed). The v2.3 production GNN path (src/gnn/,GraphPredictorV2,GINEConv) is a separate module tree with zero allele references and is not affected. GNN promotion remains deferred (GPU-gated) and no certified result in the results ledger passes through this code path.Three fixes, landed together - none alone closes the gap
1. Path resolution + fail loud on absent-or-empty. The table now resolves via a module-level
Path(__file__).resolve().parent / "mhc_pseudo_sequences.json", independent of CWD. An absent file raisesFileNotFoundError; a present-but-content-empty table (only the tracked file's own_sourcedocumentation key, or truly empty) raisesValueError- the existence check alone would not have caught a gutted-but-present file falling through the identical placeholder path.2. Package data. The wheel built from this repository shipped zero non-
.pydata files (verified:git archive HEAD-> 75 entries, none outside.dist-info/), so even a CWD-correct invocation of an installed package still hit the missing-file path. Added[tool.setuptools.package-data]("src.verify" = ["*.json"]).3. Silent mock-fallback reporting.
sestrav_evaluator.py'sevaluate_single_virusalready caught a construction failure and fell back to mock predictions - correctly - but did so silently.run_evaluation_pipeline's top-levelmetadata["use_mock_fallback"]was computed once before the per-virus loop and never updated, so a report could readuse_mock_fallback: falsewhile some virus's scores were actually mock. Added aused_mock_fallbackflag threaded through bothexceptarms (main-cohort and breakout-mutant) into each virus's own result; the pipeline now recomputes its top-level flag as an OR across every virus after the loop.Verification
The fix proven end-to-end against a real installed package, not just unit tests: built a real wheel, installed it into a fresh venv (including the
[gnn]extra), and ran from a directory with no relationship to the repo:Every guard and every fallback-flag site was individually mutation-tested - reverted in isolation, confirmed its dedicated test fails, then restored and confirmed the suite is green again:
{}logic fails 3 tests.test_dataset_raises_on_metadata_only_pseudo_seqs_table(and shows both alleles silently collapsing to the placeholder in the failure log).used_mock_fallbacksite: reverting failstest_evaluate_single_virus_reports_unplanned_fallback.used_mock_fallbacksite (a separate call site from the main-cohort one - deleting only this one left every existing test green before this PR added a test that isolates it): reverting failstest_evaluate_single_virus_reports_breakout_only_fallback.test_run_evaluation_pipeline_reports_per_virus_fallback_in_metadata.Test suite:
test_structural_gnn.py+test_sestrav_evaluator_gnn.py+test_sestrav_evaluator.py+test_hla_pseudo_sequence_tables.py: 38 passed, 2 skipped (PyG-absent-only branches).ruff check/ruff format --check/mypyclean on every touched file. Local integrity harness unchanged: 151 PASS / 0 WARN / 2 FAIL / 7 SKIP (the two FAILs are standing, unrelated owner-owned rulings).Review note
This PR went through two rounds of independent adversarial review before push. The second round found and this PR fixes: a checkable-false claim in a code comment ("zero non-.py members" - actually 6, all
.dist-info/metadata, now worded correctly), a self-contradicting overclaim (a comment said an absent table makes position/edge-index tensors allele-dependent too, when those never depend on allele identity either way - only node features and edge attributes do), two mutation-testing gaps (the breakout-arm and pipeline-recompute sites above, which the first test pass did not actually pin), the empty-table gap now closed by fix #1'sValueError, and aruff formatnit. It also confirmed the scope boundary above - thatsrc/gnn/'s production path is unaffected - which is now stated explicitly in both this description and the CHANGELOG entry rather than left to be inferred.