Skip to content

fix: resolve missing dataset and config issues in compression benchmark.#542

Open
Fahmid-Arman wants to merge 1 commit into
kubeedge:mainfrom
Fahmid-Arman:fix-compression-benchmark-deps
Open

fix: resolve missing dataset and config issues in compression benchmark.#542
Fahmid-Arman wants to merge 1 commit into
kubeedge:mainfrom
Fahmid-Arman:fix-compression-benchmark-deps

Conversation

@Fahmid-Arman

Copy link
Copy Markdown

What type of PR is this?
/kind bug

What this PR does / why we need it:
The single_task_bench_with_compression example referenced a dataset path from the unrelated government example, causing the benchmarking job to fail validation immediately. This PR completely restores the example for standalone execution.

Changes implemented:

  • Dataset Integration: Added a minimal, self-contained dataset (dataset/train_data/data.jsonl and dataset/test_data/data.jsonl) so the example runs end-to-end without requiring external downloads.
  • Path Resolution: Updated testenv/testenv.yaml to point at the new dataset using root-relative paths (./examples/...).
  • Version Control: Added a scoped negation to .gitignore so this example's dataset is tracked despite the repo-wide dataset/ ignore rule.
  • Dependency Fix: Stripped a hidden U+200E (LEFT-TO-RIGHT MARK) character from the requirements.txt filename that silently broke pip install.
  • Documentation Cleanup: Removed misleading advice in README.md requiring hardcoded absolute paths, fixed malformed wget syntax, and updated configuration steps.

Which issue(s) this PR fixes:
Fixes #249

@kubeedge-bot kubeedge-bot added kind/bug Categorizes issue or PR as related to a bug. do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. labels Jun 13, 2026
@kubeedge-bot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Fahmid-Arman
To complete the pull request process, please assign jaypume after the PR has been reviewed.
You can assign the PR to them by writing /assign @jaypume in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubeedge-bot kubeedge-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jun 13, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request bundles a minimal dataset directly with the single-task compression benchmark suite, updates the test environment configuration to use relative paths pointing to this dataset, and simplifies the setup instructions in the README. Feedback highlights path inconsistencies in the README's model download commands and a discrepancy between the execution directory assumed by the relative paths in the configuration and the commands documented in the README.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

```bash
mkdir -p /ianvs/models/qwen
wget -c -O ianvs/models/qwen/qwen_1_5_0_5b.gguf [https://huggingface.co/Qwen/Qwen1.5-0.5B-Chat-GGUF/resolve/main/qwen1_5-0_5b-chat-q4_k_m.gguf](https://huggingface.co/Qwen/Qwen1.5-0.5B-Chat-GGUF/resolve/main/qwen1_5-0_5b-chat-q4_k_m.gguf)
wget -c -O ianvs/models/qwen/qwen_1_5_0_5b.gguf https://huggingface.co/Qwen/Qwen1.5-0.5B-Chat-GGUF/resolve/main/qwen1_5-0_5b-chat-q4_k_m.gguf

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

There is a path inconsistency between the directory creation command and the wget download command:

  1. mkdir -p /ianvs/models/qwen (on line 30) uses an absolute path starting with /, which might fail due to permission issues and is likely not the intended location.
  2. wget -c -O ianvs/models/qwen/qwen_1_5_0_5b.gguf (on line 31) uses a relative path starting with ianvs/models/....

Additionally, .gitignore (line 8) ignores models/qwen/ at the repository root. If the benchmark expects the model to be in the repository's root models/qwen/ directory, both commands should be updated to use models/qwen/ instead of /ianvs/models/qwen or ianvs/models/qwen.

Please update the wget command to use the correct relative path models/qwen/qwen_1_5_0_5b.gguf.

Suggested change
wget -c -O ianvs/models/qwen/qwen_1_5_0_5b.gguf https://huggingface.co/Qwen/Qwen1.5-0.5B-Chat-GGUF/resolve/main/qwen1_5-0_5b-chat-q4_k_m.gguf
wget -c -O models/qwen/qwen_1_5_0_5b.gguf https://huggingface.co/Qwen/Qwen1.5-0.5B-Chat-GGUF/resolve/main/qwen1_5-0_5b-chat-q4_k_m.gguf

Comment on lines +3 to +4
train_data: "./examples/llm-edge-benchmark-suite/single_task_bench_with_compression/dataset/train_data/data.jsonl"
test_data: "./examples/llm-edge-benchmark-suite/single_task_bench_with_compression/dataset/test_data/data.jsonl"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

These root-relative paths (./examples/...) assume that the ianvs command is executed from the repository root directory. However, the execution commands provided in the README.md (e.g., ianvs -f ianvs/examples/... and pip install -r ianvs/examples/...) prepend ianvs/ to the paths, implying execution from the parent directory of the repository.

To prevent path resolution failures, please ensure the documentation in README.md is updated to remove the ianvs/ prefix from the commands, so that users run them from the repository root where these relative paths will resolve correctly.

@Fahmid-Arman Fahmid-Arman force-pushed the fix-compression-benchmark-deps branch from 885da8a to 73787c7 Compare June 13, 2026 15:41
@kubeedge-bot kubeedge-bot removed the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Jun 13, 2026
…rk example

Signed-off-by: Fahmid Arman <fahmid.brac@gmail.com>
@Fahmid-Arman Fahmid-Arman force-pushed the fix-compression-benchmark-deps branch from 73787c7 to 32b6290 Compare June 13, 2026 15:45
@Fahmid-Arman

Copy link
Copy Markdown
Author

Feedback addressed. All ianvs/ and absolute path prefixes removed from README.md execution and download commands to align with repository-root execution. Associated pip install prefix inconsistency also resolved.

/assign @jaypume

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Categorizes issue or PR as related to a bug. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PRE-TEST: llm-edge-benchmark-suite/single_task_bench_with_compression references missing dataset in testenv.yaml

3 participants