fix(python): isolate the torch.compile cache per xdist worker - #9355
jackylee-ch wants to merge 1 commit into
Conversation
`test_torch_index_with_nans` fails intermittently on the Windows python job with `PermissionError: [Errno 13]` on a path under `torchinductor_runneradmin`. It has now reproduced on two consecutive runs of the same branch, once per parametrized case, so it is not a one-off. `lance.torch.distance` decorates six kernels with `@torch.compile`, the suite runs with `PYTEST_WORKERS=auto`, and nothing sets `TORCHINDUCTOR_CACHE_DIR`. Every worker that reaches the torch accelerator therefore compiles the same six functions into torch's single default cache directory. Inductor writes a generated module by replacing it, which POSIX permits while another process holds it open and Windows refuses -- which is why only this job is affected. Point each worker at its own subdirectory. Serial runs keep torch's default so the cache stays warm, and the per-worker directories share one temp root for the same reason. `TORCH_COMPILE_DISABLE` would also stop the race but would drop coverage of those six kernels, which the Windows job sets up MSVC specifically to exercise. ## Testing `uv run make lint` from `python/`. Verified the mechanism with a throwaway test asserting that each worker's `TORCHINDUCTOR_CACHE_DIR` ends in its own worker id: under `-n 2` the two workers get `.../lance_torchinductor/gw0` and `.../gw1`, and with the helper disabled the variable is unset for both. I cannot reproduce the Windows failure itself on macOS, so that the race is what fails there remains inferred from the log and the configuration above.
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
There was a problem hiding this comment.
❌ Gate recommendation: request changes.
The per-worker cache strategy addresses the Windows xdist race and preserves serial runs and explicit cache roots. Please make the fix merge-ready by adding durable, lightweight regression coverage for the helper mapping; reproducing Windows file locking is unnecessary.
| ) | ||
|
|
||
| workerinput = getattr(config, "workerinput", None) | ||
| _isolate_torch_inductor_cache(workerinput) |
There was a problem hiding this comment.
This bug fix has no committed regression test, so the worker isolation can regress without detection and the current head does not meet the repository requirement that every bug fix have corresponding tests. Please add a lightweight test for _isolate_torch_inductor_cache covering None (serial execution leaves the environment unchanged), two worker IDs (distinct child directories), and a pre-set cache root. Those mapping cases exercise the contract directly without needing to reproduce Windows file locking.
lance-gatekeeper
Bot
added
the
K-changes
1 participant
test_torch_index_with_nansfails intermittently on the Windows python job withPermissionError: [Errno 13]undertorchinductor_runneradmin, now on two consecutive runs of one branch.lance.torch.distancedecorates six kernels with@torch.compile, the suite runs withPYTEST_WORKERS=auto, and nothing setsTORCHINDUCTOR_CACHE_DIR, so every worker reaching the torch accelerator compiles those six into torch's single default cache directory. Inductor writes a generated module by replacing it, which POSIX permits while another process holds it open and Windows refuses -- hence only this job.Point each worker at its own subdirectory; serial runs keep torch's default.
Testing
uv run make lintfrompython/. A throwaway test asserting each worker'sTORCHINDUCTOR_CACHE_DIRends in its worker id passes under-n 2and fails with the helper disabled. I cannot reproduce the Windows failure on macOS, so the race being the cause stays inferred.#8655 also adds an
os.environline at the top ofpytest_configure; these two will conflict.