Skip to content

fix: remove worktree from PYTHONPATH to prevent system package shadowing#97

Open
iraj465 wants to merge 1 commit into
mainfrom
fix/eval-pythonpath-shadowing
Open

fix: remove worktree from PYTHONPATH to prevent system package shadowing#97
iraj465 wants to merge 1 commit into
mainfrom
fix/eval-pythonpath-shadowing

Conversation

@iraj465
Copy link
Copy Markdown
Collaborator

@iraj465 iraj465 commented Apr 1, 2026

During optimization, agents may create directories in their worktree that match system package names (e.g. aiter/). Because GEAK_WORK_DIR was placed first in PYTHONPATH, these shallow stubs shadowed the real system packages, causing ImportError during full_benchmark evaluation.

Remove GEAK_WORK_DIR from PYTHONPATH in both the COMMANDMENT SETUP template (run.sh / run_harness.sh) and build_eval_env(). The worktree is not needed on PYTHONPATH: harnesses load kernel.py via importlib with an explicit file path, and Python automatically adds the script directory to sys.path[0].

Example:

  1. Preprocessing — GEAK sets up the kernel workspace:

worktree: /path/to/worktrees/slot_0/
├── kernel.py (the triton kernel to optimize)
├── test_kernel_harness.py
└── (nothing else)
2. Agent optimizes — the AI agent edits kernel.py, runs save_and_test. During optimization, the agent sometimes creates helper files. For fused_append_shared_experts, the kernel imports from sglang which imports from aiter. The agent sees import errors, tries to "fix" them by creating a local aiter/ package stub:

worktree: /path/to/worktrees/slot_0/
├── kernel.py
├── test_kernel_harness.py
└── aiter/ ← agent created this
├── init.py ← stub: just "from pkgutil import extend_path"
└── ops/
3. Round ends, full_benchmark runs — GEAK's evaluation.py runs the harness in a subprocess with:

export PYTHONPATH=/path/to/worktrees/slot_0:/repo_root:$PYTHONPATH
python3 test_kernel_harness.py --full-benchmark
4. Python resolves import aiter — it searches PYTHONPATH left to right:

/path/to/worktrees/slot_0/aiter/init.py ← FOUND FIRST (the stub!)
/sgl-workspace/aiter/aiter/init.py ← never reached
5. Import fails:

from aiter import dynamic_per_tensor_quant # stub doesn't have this → ImportError
Would we need this fix for any new kernel URL?
Yes, but only if the agent creates files that shadow a system package. The issue is general:

GEAK puts the worktree first in PYTHONPATH
If the agent creates any directory that matches a system package name (e.g., torch/, numpy/, triton/, aiter/), it will shadow the real package
This isn't aiter-specific — it could happen with any package
In practice, it happens most with aiter because:

Many kernels import from sglang → sglang imports aiter
The agent sees aiter import errors and tries to create a local package to "fix" them
But any kernel where the agent creates a directory matching a system package would hit this
The fix is general-purpose: removing the worktree from PYTHONPATH (or putting it last) prevents ANY package shadowing, not just aiter. The harness loads kernel.py via importlib with an explicit file path — it doesn't need PYTHONPATH to find it. So removing the worktree from PYTHONPATH has no downside

@iraj465 iraj465 force-pushed the fix/eval-pythonpath-shadowing branch from eb02b58 to 3c7d621 Compare April 1, 2026 17:53
Two fixes to prevent agent-created directories from shadowing system
packages (e.g. a stub aiter/ shadowing /sgl-workspace/aiter/):

1. Remove GEAK_WORK_DIR from PYTHONPATH in build_eval_env() and all
   COMMANDMENT SETUP templates.  The worktree is not needed on
   PYTHONPATH since harnesses load kernel.py via importlib with an
   explicit file path.

2. Clean agent-created package directories from the eval worktree
   before running full_benchmark.  Python adds the script directory to
   sys.path[0] automatically, so any __init__.py there would still
   shadow system packages even without PYTHONPATH.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@iraj465 iraj465 force-pushed the fix/eval-pythonpath-shadowing branch from 3c7d621 to 0d3f3bc Compare April 1, 2026 18:27
@sdubagun-amd
Copy link
Copy Markdown
Collaborator

sdubagun-amd commented Apr 8, 2026

@iraj465 Can you update the PR after addressing the merge conflicts and CI failures?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants