Record how the platform-height correction fakes a 31% win - #8
Merged
Merged
Conversation
The merged note says a platform-height correction to shotL/shotR buys 3% on arc height at the rim and is not worth shipping. A second session of flights has since made that conclusion much stronger, and turned up the trap that nearly reversed it. Pooling the two sessions makes the same correction look like a 31% win, 75.2px down to 51.8px. Split by session: session A (n=8) 54.7px -> 53.0px -3% session B (n=7) 61.4px -> 75.1px +23% WORSE pooled (n=15) 75.2px -> 51.8px -31% It helps marginally in one session, hurts materially in the other, and only looks good pooled. The tell is that pooling made the CONSTANT model worse than it was in either session alone, which can only happen if pooling introduced variance neither session had; the extra free parameter then absorbs it and leave-one-out rewards it for that rather than for predicting anything. There is no real drift to absorb either: per session the parameters agree to within .21 of a standard deviation, so calibration does carry across sessions and the pooled variance was not a genuine offset. No behaviour change. This exists so the next person who notices the platform-height correlation -- it is strong, and it replicates -- does not pool their sessions, see 31%, and ship it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #7, numbers only, no behaviour change.
#7 concluded that a platform-height correction to
shotL/shotRbuys 3% on arc height at the rim and isn't worth shipping. A second session of flights has since made that conclusion stronger — and turned up the trap that nearly reversed it.Pooling the two sessions makes the same correction look like a 31% win:
It helps marginally in one session, hurts materially in the other, and only looks good pooled. The tell: pooling made the constant model worse than it was in either session alone (75.2 vs 54.7 and 61.4), which can only happen if pooling introduced variance neither session had. The extra free parameter absorbs that, and leave-one-out rewards it for absorbing an artefact rather than for predicting anything.
There's no real drift to model either — per session the parameters agree to within 0.21 of a standard deviation, so calibration carries across sessions fine.
This exists so the next person who notices the platform-height correlation (it's strong, and it replicates at −0.78) doesn't pool their sessions, see 31%, and ship it.