Make code_eval reliable in fleets: pin filelock and isolate metrics cache per batch

[maxidl]

Two independent fixes, both required for the HF evaluate-based code tasks (HumanEval / MBPP / LBPP + their deu variants) to run in a multi-job eval-hive fleet. Both are latent bugs that only surface under fork + concurrency, so single/serial runs mask them.

1. Pin filelock < 3.16 (pixi.toml)

filelock >= 3.16 adds an at-fork audit that raises

RuntimeError: os.fork is unsafe while filelock is changing descriptor ownership

when code_eval forks to execute generated code. This kills every code task regardless of concurrency.

2. Isolate evaluate's metrics cache per batch (create_run.py)

evaluate names its compute-cache file only by experiment_id:

$HF_METRICS_CACHE/code_eval/default/default_experiment-1-0.arrow

and HF_METRICS_CACHE defaults to $HF_HOME/metrics (usually shared storage). Concurrent batches — parallel_tasks > 1, and across jobs that share HF_HOME — race on that single file: one process finishes and os.remove()s it, the next crashes with FileNotFoundError, failing the whole batch (up to task_batch_size tasks) as collateral. The batch runner now sets a per-batch node-local HF_METRICS_CACHE.

Verification

Hit both on a 31-checkpoint prelude fleet (30 nodes sharing one gpfs HF_HOME). Symptoms: code tasks crashed with the os.fork error (fix 1), then — after pinning filelock — with the default_experiment FileNotFoundError (fix 2); ~2 code-containing batches failed per job → ~15 missing tasks per checkpoint and no code pass@1. With both fixes, a 3-job canary showed 0 race errors and all code tasks landed, and the full fleet then completed 332/332 tasks on every checkpoint.

🤖 Generated with Claude Code