feat(deps): add rouge extra for ROUGE scoring deps - #506
Open
anandhu-eng wants to merge 1 commit into
Open
anandhu-eng wants to merge 1 commit into
anandhu-eng wants to merge 1 commit into
Conversation
`RougeScorer` (eval_method: "rouge") lazily imports nltk, evaluate and
rouge_score and raises ImportError telling users to pip install them, but
no extra ever declared them. The three shipped Llama configs that use
eval_method: "rouge" therefore fail at scorer construction on a stock
install; the only instructions lived in examples/05_Llama_Examples/README.md.
Declare them in a new `rouge` extra and have the `test` extra inherit it
(same self-reference pattern as `inference-endpoint[sql]`), so CI covers
the import surface with no extra job. Resolution is clean: 4 new packages,
`evaluate` pulls the already-pinned datasets==5.0.1, and no existing pin
moves - so unlike bfcl this needs no [tool.uv].conflicts fork.
Not self-contained at runtime: nltk needs the punkt/punkt_tab corpora and
evaluate.load("rouge") fetches its metric script from the HF Hub. Offline
images must prefetch both - handled separately in Dockerfile.dev.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
the
size/normal
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
RougeScorer(eval_method: "rouge") lazily importsnltk,evaluateandrouge_scoreand raises anImportErrortelling users to install them — but no extra ever declared them. The three shipped Llama configs usingeval_method: "rouge"fail at scorer construction on a stock install, and the only install instructions lived inexamples/05_Llama_Examples/README.md.Change
rougeextra declaringnltk==3.10.3,evaluate==0.4.6,rouge-score==0.1.2.testextra inherits it viainference-endpoint[rouge]— same self-reference pattern asinference-endpoint[sql]— so CI covers the import surface with no extra job.Resolution is clean: 4 new packages,
evaluatepulls the already-pinneddatasets==5.0.1, and no existing pin moves. Unlikebfcl, no[tool.uv].conflictsfork is needed.Verified
uv lockadds onlyabsl-py,evaluate,nltk,rouge-score; no version changes to existing packages.uv sync --frozen --extra rougesucceeds; all three import at the pinned versions.evaluation/scoring.pynow resolves (_evaluate/_nltknon-None), so theImportErrorpath is no longer hit.Note
Installing the packages is not sufficient for an offline image:
nltkneeds thepunkt/punkt_tabcorpora andevaluate.load("rouge")fetches its metric script from the HF Hub. Prefetching both is handled separately inDockerfile.dev.🤖 Generated with Claude Code