fix: remove worktree from PYTHONPATH to prevent system package shadowing#97
Open
iraj465 wants to merge 1 commit into
Open
fix: remove worktree from PYTHONPATH to prevent system package shadowing#97iraj465 wants to merge 1 commit into
iraj465 wants to merge 1 commit into
Conversation
eb02b58 to
3c7d621
Compare
Two fixes to prevent agent-created directories from shadowing system packages (e.g. a stub aiter/ shadowing /sgl-workspace/aiter/): 1. Remove GEAK_WORK_DIR from PYTHONPATH in build_eval_env() and all COMMANDMENT SETUP templates. The worktree is not needed on PYTHONPATH since harnesses load kernel.py via importlib with an explicit file path. 2. Clean agent-created package directories from the eval worktree before running full_benchmark. Python adds the script directory to sys.path[0] automatically, so any __init__.py there would still shadow system packages even without PYTHONPATH. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
3c7d621 to
0d3f3bc
Compare
Collaborator
|
@iraj465 Can you update the PR after addressing the merge conflicts and CI failures? |
0d3f3bc to
444bfd3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During optimization, agents may create directories in their worktree that match system package names (e.g. aiter/). Because GEAK_WORK_DIR was placed first in PYTHONPATH, these shallow stubs shadowed the real system packages, causing ImportError during full_benchmark evaluation.
Remove GEAK_WORK_DIR from PYTHONPATH in both the COMMANDMENT SETUP template (run.sh / run_harness.sh) and build_eval_env(). The worktree is not needed on PYTHONPATH: harnesses load kernel.py via importlib with an explicit file path, and Python automatically adds the script directory to sys.path[0].
Example:
worktree: /path/to/worktrees/slot_0/
├── kernel.py (the triton kernel to optimize)
├── test_kernel_harness.py
└── (nothing else)
2. Agent optimizes — the AI agent edits kernel.py, runs save_and_test. During optimization, the agent sometimes creates helper files. For fused_append_shared_experts, the kernel imports from sglang which imports from aiter. The agent sees import errors, tries to "fix" them by creating a local aiter/ package stub:
worktree: /path/to/worktrees/slot_0/
├── kernel.py
├── test_kernel_harness.py
└── aiter/ ← agent created this
├── init.py ← stub: just "from pkgutil import extend_path"
└── ops/
3. Round ends, full_benchmark runs — GEAK's evaluation.py runs the harness in a subprocess with:
export PYTHONPATH=/path/to/worktrees/slot_0:/repo_root:$PYTHONPATH
python3 test_kernel_harness.py --full-benchmark
4. Python resolves import aiter — it searches PYTHONPATH left to right:
/path/to/worktrees/slot_0/aiter/init.py ← FOUND FIRST (the stub!)
/sgl-workspace/aiter/aiter/init.py ← never reached
5. Import fails:
from aiter import dynamic_per_tensor_quant # stub doesn't have this → ImportError
Would we need this fix for any new kernel URL?
Yes, but only if the agent creates files that shadow a system package. The issue is general:
GEAK puts the worktree first in PYTHONPATH
If the agent creates any directory that matches a system package name (e.g., torch/, numpy/, triton/, aiter/), it will shadow the real package
This isn't aiter-specific — it could happen with any package
In practice, it happens most with aiter because:
Many kernels import from sglang → sglang imports aiter
The agent sees aiter import errors and tries to create a local package to "fix" them
But any kernel where the agent creates a directory matching a system package would hit this
The fix is general-purpose: removing the worktree from PYTHONPATH (or putting it last) prevents ANY package shadowing, not just aiter. The harness loads kernel.py via importlib with an explicit file path — it doesn't need PYTHONPATH to find it. So removing the worktree from PYTHONPATH has no downside