feat(docker): provision ROUGE scorer deps and corpora in dev image - #507
Open
anandhu-eng wants to merge 1 commit into
Open
anandhu-eng wants to merge 1 commit into
anandhu-eng wants to merge 1 commit into
Conversation
Install the `rouge` extra in both uv sync layers and prefetch the scorer's
runtime data, so a pushed image can run `eval_method: "rouge"` with no
network - matching how PROVISION_DSR1 / PROVISION_VBENCH already make the
DeepSeek-R1 and VBench evaluators self-contained.
The extra is declared explicitly rather than inherited from `test`, since
ROUGE is a runtime capability of the published image and should not depend
on a test-only extra to supply it.
Installing the packages is not enough on its own: nltk.sent_tokenize needs
the punkt/punkt_tab corpora and evaluate.load("rouge") fetches its metric
script from the HF Hub. A new PROVISION_ROUGE=1 block prefetches both and
then asserts the corpora resolve, so a silent download failure fails the
build instead of the benchmark.
Both caches land in their default per-user locations (~/nltk_data,
~/.cache/huggingface). Deliberately not redirected to /opt via
NLTK_DATA/HF_HOME: HF_HOME is global at runtime, so repointing it would
orphan what the DSR1/VBench stages already cached under the default path.
The block sits after VBench and before the LABEL layer, so editing it
rebuilds neither the heavy evaluator stages nor anything but image config.
Requires the `rouge` extra from #506 - merge that first, or `uv sync
--extra rouge` fails here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
github-actions
Bot
added
the
size/normal
attafosu
approved these changes
Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
rougeextra in bothuv synclayers, so the published image can runeval_method: "rouge". Declared explicitly rather than inherited fromtest— ROUGE is a runtime capability of the image and shouldn't depend on a test-only extra.PROVISION_ROUGE=1block prefetching the scorer's runtime data, matching howPROVISION_DSR1/PROVISION_VBENCHalready make their evaluators self-contained.Installing the packages isn't enough on its own:
nltk.sent_tokenizeneeds thepunkt/punkt_tabcorpora andevaluate.load("rouge")fetches its metric script from the HF Hub. The block prefetches both, then asserts the corpora resolve — a silent download failure fails the build rather than the benchmark.Two deliberate choices
Caches stay in their defaults (
~/nltk_data,~/.cache/huggingface) rather than/optviaNLTK_DATA/HF_HOME.HF_HOMEis global at runtime, so repointing it would orphan whatever the DSR1/VBench stages already cached under the default path.Placed after VBench, before
LABEL— editing it rebuilds neither the heavy evaluator stages above nor anything but image config below.Verified
A full build of this file isn't possible until #506 merges, so the block was validated in an isolated
python:3.12.11-slimprobe image running as a non-root user with the identical commands:rouge metric cached+punkt corpora ok.docker run --network=nonecomputesrougeLfrom the baked cache — the offline claim, confirmed end to end.PROVISION_ROUGE=0correctly skips the block.Probe images were removed afterward. The prefetch commands were also run against the real
rouge-synced venv, including an offline replay.🤖 Generated with Claude Code