Fix region-matching id collision; re-score end-to-end systems (v0.1.7.2)#26
Merged
Merged
Conversation
match_regions honored an exact region_id string match before checking
spatial overlap. Both end-to-end runners emitted 0-indexed ids
(r0, r1, ...) that collide with the ground-truth r1, r2, ... namespace,
so end-to-end regions were paired off-by-one to non-overlapping
ground-truth regions, corrupting chrF, IoU, tau and LTB-100 for every
end-to-end system. Oracle-layout and identity systems were unaffected.
- match_regions: exact-id matches gated on IoU >= 0.10; a coincidental
id collision now falls through to greedy IoU matching
- end-to-end runners emit non-colliding p{i} ids; qwen3-vl submissions
re-indexed
- score Florence-NLLB end-to-end (Florence-2 OCR + NLLB-200):
LTB-100 39.79 chrF / 48.11 COMET-Kiwi, 8/16 pairs
- re-score qwen3-vl: 14.82 chrF / 18.43 COMET-Kiwi
- use short system display names on the leaderboard; full model ids
remain in model_id_or_url
- regenerate leaderboard; fix the generator's stale v0.1.6 sample-size
caveat and __version__
- methodology roadmap: v0.1.7.2 entry; critique #9 figures corrected
84 tests pass.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
match_regionshonored an exactregion_idstring match before checking spatial overlap. Both end-to-end runners (Florence-NLLB, Qwen3-VL) emitted 0-indexed ids (r0, r1, ...) that collide with the ground-truthr1, r2, ...namespace, so end-to-end regions were force-paired off-by-one to non-overlapping ground-truth regions — corrupting chrF, IoU, τ and LTB-100 for every end-to-end system. Oracle-layout and identity systems were unaffected (they copy ground-truth ids and boxes).Changes
match_regions): an exact-id match is now honored only when the boxes also overlap (IoU ≥ 0.10); a coincidental id collision falls through to greedy IoU matching.p{i}ids; committed qwen3-vl submissions re-indexed.__version__.84 tests pass.
Corrected end-to-end leaderboard
Oracle-layout systems (DeepL, NLLB, opus-mt) unchanged — never affected by the bug.