Fix confidence scorer loader misrouting joblib artifacts#8
Merged
Conversation
load_lightgbm_scorer routed by whether metadata_path was passed: any metadata_path (explicit or auto-discovered sidecar) forced the raw LightGBM Booster loader. But train_confidence_scorer exports a self-contained joblib dict and also offers write_metadata_json, so a caller with both files who passes metadata_path (exactly what spec_confidence_runtime_readiness's from_artifact documents) got 'Unknown model format' on the pipeline's own artifact. Route by artifact content instead: try joblib.load first; fall back to the Booster loader only when the file is not a joblib pickle. metadata stays authoritative from the joblib dict; the sidecar/metadata_path is used only for raw Booster files that cannot carry it. Adds a regression test loading a joblib dict WITH metadata_path. Verified end-to-end on a real trained scorer.joblib. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pko89403
pushed a commit
that referenced
this pull request
Jul 16, 2026
Review confirmed the docs contradicted the code this branch ships: - 05_open_questions.md: the Q004 entry sat inside the '## 형식' template code fence, so it rendered as example code instead of a real section. It is now a real, resolved entry recording the final decisions (new Strategy, raw-confidence descending sort, QA/SQuAD training data) and the fate of the smoke-run blockers (loader bug fixed via PR #8). - spec_confidence_aware_reranking.md: §2/§3/§5/§6 presented the draft judgment_confidence + LCR binning/gate design as current requirements; they are now explicitly marked superseded with the shipped deltas, §7 marks all four Q004 decisions resolved, and the task checklist matches reality (implementation done, README-table run remaining). - README.md / README.ko.md: dropped the acc@1/MRR numbers the spec forbids putting in the README and the false "committed evaluation" provenance (no evidence artifact exists in the repo); the honest qualitative warning and pointers remain. - 02_architecture.md: strategy list, ModelClient contracts, file tree, and JSON response contracts now include the answer path; same provenance correction as the README. - spec_confidence_generation_pipeline.md: no_answer_value is documented as the shared ranksmith.model.NO_ANSWER_VALUE constant instead of the removed config field. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011d5n8TDf8vg22SZYggjnBb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
load_lightgbm_scorerdecided the artifact format by whethermetadata_pathwas passed, not by what the artifact actually is. Anymetadata_path— explicit, or an auto-discovered.metadata.jsonsidecar — forced the raw-LightGBM-Booster loader (lightgbm.Booster(model_file=...)).But
train_confidence_scorerexports a self-contained joblib dict (joblib.dump({"metadata":..., "scorer":...})) and offerswrite_metadata_jsonto write a sidecar. So a caller who has both files and passesmetadata_path— which is exactly whatspec_confidence_runtime_readiness.md'sfrom_artifact(path, metadata_path=...)documents — gotLightGBMError: Unknown model formaton the pipeline's own artifact.This was invisible until now because no trained artifact was committed and nothing ran train→export→load on one real file. It surfaced while producing a real scorer from BEIR/SciFact data via the training pipeline. Same root cause as the earlier YAGNI review finding (the loader supports four artifact formats but the pipeline produces exactly one).
Fix
Route by artifact content: try
joblib.loadfirst; fall back to the Booster loader only when the file is not a joblib pickle. Metadata stays authoritative from the joblib dict; the sidecar /metadata_pathis used only for raw Booster files that cannot carry metadata themselves. Existing Booster+sidecar behavior is preserved.Adds a regression test that loads a joblib dict with
metadata_path(the exact call that used to fail). Verified end-to-end on a real trainedscorer.joblib: the previously-failingload_lightgbm_scorer(path, metadata_path=...)now returns a workingJoblibScorerWrapperand scores held-out features.432 passed(was 431 + 1 regression test), mypy + ruff clean.🤖 Generated with Claude Code