fix: resolve execution blockers and hardcoded paths in llm_simple_qa benchmark#544
fix: resolve execution blockers and hardcoded paths in llm_simple_qa benchmark#544Fahmid-Arman wants to merge 1 commit into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Fahmid-Arman The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Code Review
This pull request updates the llm_simple_qa example by replacing hardcoded local paths with relative ones, adding a dataset preparation script (prepare_data.py), and updating basemodel.py to dynamically select the device and load the public Qwen/Qwen2-0.5B-Instruct model. The review feedback suggests enhancing device selection to support Apple Silicon (mps), explicitly setting torch.float32 on CPU to prevent half-precision runtime crashes, and adding a safety check in prepare_data.py to avoid a FileNotFoundError when creating directories.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
…benchmark Signed-off-by: Fahmid Arman <fahmid.brac@gmail.com>
eb6a48b to
9af62d9
Compare
|
Feedback addressed.
/assign @jaypume |
What type of PR is this?
/kind bug
What this PR does / why we need it:
The
llm_simple_qaexample previously contained multiple execution-blocking bugs, including hardcoded local environment paths, missing framework interfaces, and missing dataset automation. These issues prevented users from executing the benchmark out-of-the-box.Changes implemented:
/home/icyfeather/...) and stale relative paths (./examples/llm/...) with accurate, portable root-relative paths inbenchmarkingjob.yaml,testenv.yaml, andgen_algorithm.yaml.prepare_data.pyscript to automatically generate the required JSONL data structure."Qwen/Qwen2-0.5B-Instruct"). Replaced hardcodedcudaallocation with dynamic hardware detection ("cuda" if torch.cuda.is_available() else "cpu") to prevent immediate crashes on non-Nvidia/CPU-only machines.preprocesspass-through method tobasemodel.pyto satisfy thesingletasklearningparadigm requirements.sedna-llm.zip) and updated the execution paths inREADME.md.Which issue(s) this PR fixes:
Fixes #368