Eval framework + 4090-campaign tooling (pre-registration, r²aff readout, DANN de-classification) - #2
Open
KoushikVGitHub wants to merge 3 commits into
Open
Eval framework + 4090-campaign tooling (pre-registration, r²aff readout, DANN de-classification)#2KoushikVGitHub wants to merge 3 commits into
KoushikVGitHub wants to merge 3 commits into
Conversation
Files-as-truth run manifests, pre-registered hash-chained claims (git as the tamper-evident ledger), and a deterministic judge whose gate encodes the design review must-fixes: UCB run-to-run SD noise band + reported MDE (S2), floor parity (S5), split hygiene (S6), pre-registration (S1). No LLM in the numeric path. Dogfoods the DOF grounding study end-to-end: certifies the pk-anchor (0.833) and data-manifold-ID claims, and abandons the suite-separability claim whose negative control caught train-set overfitting (CV 0.588, not the inflated 0.79). Adds the DOF study (scripts/dof_probe.py) + its reproducible confirmation harness (scripts/dof_tests.py) and the mfs concurrency/crash probes that validated files-as-truth on the network volume. CI (.github/workflows/eval-ci.yml): CPU-only, no torch, no data.
…ation (suite + feedback GRL), pre-registration ledger
- scripts/ridge_readout.py: pure-numpy ridge + r2aff transfer readout control (no sklearn)
- src/declassify.py: gradient-reversal + suite/feedback adversaries (Miest-style suite-invariance)
- src/{jepa_loss,train_fsdp}.py: expose pooled features + wire invariance-mode training
- eval/claims/astrid_transfer.jsonl: hash-chained pre-registered claims (GRL levers; verified chain)
- eval/preregister_*.py: register/abandon the Astrid->SIMBA claim set
Result recorded in L22: both invariance levers are NULL for cross-suite transfer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the evaluation framework and the RTX-4090 DOF-campaign tooling onto
main.What this adds
eval/— files-as-truth eval harness: hash-chained pre-registered claims (lib/claims.py), a statistical gate with UCB noise-band + MDE (lib/gate.py), run tracing (lib/trace.py), and thejudge.eval/claims/astrid_transfer.jsonl— the pre-registered claim ledger for the invariance experiment (chain-verified, 6 active claims).scripts/ridge_readout.py— pure-numpy Ridge + r²aff transfer readout control (separates encoder information from readout calibration; no sklearn dependency).src/declassify.py— gradient-reversal + suite/feedback adversaries (Miest-style suite-invariance) and thetrain_fsdp.py/jepa_loss.pywiring to train with an invariance objective.Honest result (see
learnings.mdL22)Both invariance levers — suite de-classification and feedback-parameter invariance — are NULL for cross-suite transfer. The pre-registered claims are recorded as such. This PR publishes the tooling and the pre-registration, not a positive result.
Notes for the reviewer
eval-frameworkpredates L20–L22 onmain; merge direction is intomain(no conflicts —mainnever touchedtrain_fsdp.py/jepa_loss.py, and this branch never touchedlearnings.md)..bak_predeclassifybackups were intentionally excluded.