Skip to content

fix: resolve execution blockers and hardcoded paths in llm_simple_qa benchmark#544

Open
Fahmid-Arman wants to merge 1 commit into
kubeedge:mainfrom
Fahmid-Arman:fix-llm-simple-qa
Open

fix: resolve execution blockers and hardcoded paths in llm_simple_qa benchmark#544
Fahmid-Arman wants to merge 1 commit into
kubeedge:mainfrom
Fahmid-Arman:fix-llm-simple-qa

Conversation

@Fahmid-Arman

Copy link
Copy Markdown

What type of PR is this?
/kind bug

What this PR does / why we need it:
The llm_simple_qa example previously contained multiple execution-blocking bugs, including hardcoded local environment paths, missing framework interfaces, and missing dataset automation. These issues prevented users from executing the benchmark out-of-the-box.

Changes implemented:

  • Path Remediation: Replaced all hardcoded absolute paths (/home/icyfeather/...) and stale relative paths (./examples/llm/...) with accurate, portable root-relative paths in benchmarkingjob.yaml, testenv.yaml, and gen_algorithm.yaml.
  • Dataset Automation: Added a prepare_data.py script to automatically generate the required JSONL data structure.
  • Hardware Abstraction & Model Loading: Replaced the local model path with the Hugging Face Hub ID ("Qwen/Qwen2-0.5B-Instruct"). Replaced hardcoded cuda allocation with dynamic hardware detection ("cuda" if torch.cuda.is_available() else "cpu") to prevent immediate crashes on non-Nvidia/CPU-only machines.
  • Framework Integration: Added the missing preprocess pass-through method to basemodel.py to satisfy the singletasklearning paradigm requirements.
  • Documentation: Corrected the referenced Sedna zip file name (sedna-llm.zip) and updated the execution paths in README.md.

Which issue(s) this PR fixes:
Fixes #368

@kubeedge-bot kubeedge-bot added the kind/bug Categorizes issue or PR as related to a bug. label Jun 14, 2026
@kubeedge-bot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Fahmid-Arman
To complete the pull request process, please assign jaypume after the PR has been reviewed.
You can assign the PR to them by writing /assign @jaypume in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubeedge-bot kubeedge-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 14, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the llm_simple_qa example by replacing hardcoded local paths with relative ones, adding a dataset preparation script (prepare_data.py), and updating basemodel.py to dynamically select the device and load the public Qwen/Qwen2-0.5B-Instruct model. The review feedback suggests enhancing device selection to support Apple Silicon (mps), explicitly setting torch.float32 on CPU to prevent half-precision runtime crashes, and adding a safety check in prepare_data.py to avoid a FileNotFoundError when creating directories.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread examples/llm_simple_qa/testalgorithms/gen/basemodel.py
Comment thread examples/llm_simple_qa/testalgorithms/gen/basemodel.py Outdated
Comment thread examples/llm_simple_qa/prepare_data.py Outdated
…benchmark

Signed-off-by: Fahmid Arman <fahmid.brac@gmail.com>
@Fahmid-Arman

Copy link
Copy Markdown
Author

Feedback addressed.

  • Added MPS support for Apple Silicon hardware.
  • Enforced float32 precision for CPU execution to prevent Half backend errors.
  • Added empty-string validation for directory creation in prepare_data.py.

/assign @jaypume

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Categorizes issue or PR as related to a bug. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[PRE-Task 1 for LFX Mentorship 2026 Term-1]: Issue Recognition: llm_simple_qa Example Has 7 Critical Bugs Preventing Execution

3 participants