--index url biting us again

[jeanschmidt]

issue mentioned here:

https://chat.google.com/room/AAQA90pzEQs/WJMB5k5I6ic/WJMB5k5I6ic?cls=10

download.pytorch.org redirect to files.pythonhosted.org

[huydhn]

Root cause + why it's flaky (not the index URL alone)

Two things are stacked here. The index-URL redirect is the trigger; the cache-enforcer is why it's intermittent.

1. The trigger: a transitive dep that only resolves to files.pythonhosted.org

torchcomms pins torch, and that torch nightly has an unconditional dep on spmd-types==0.2.1. The nightly index doesn't host it — it re-exports the listing with hrefs pointing straight at files.pythonhosted.org:

$ curl -s https://download.pytorch.org/whl/nightly/cu130/spmd-types/
  href="https://files.pythonhosted.org/packages/20/da/.../spmd_types-0.2.1-py3-none-any.whl#sha256=..."

The install command uses --index-url https://download.pytorch.org/whl/nightly/cu130, which replaces the OSDC default PIP_INDEX_URL (the cache /simple/ proxy). The only extra index left is .../whl/cpu/, which has no spmd-types. So pip's only candidate is the pythonhosted URL → cache-enforcer RSTs it → ConnectionResetError(104).

2. Why it's flaky: cache-enforcer is defeated by kube-proxy's resync

Inspected live ip6tables on a runner node (ip-10-4-166-48, IPv6 pod). The CACHE_ENFORCER chain is fully present (v4 + v6, incl. files.pythonhosted.org) — but every counter is 0, i.e. nothing ever matches it:

-A FORWARD ... -j KUBE-FORWARD          # <- -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD ... -j KUBE-SERVICES
-A FORWARD -j CACHE_ENFORCER            # <- 4th, not 1st

The SNI string only appears in the TLS ClientHello, a data packet sent when the connection is already ESTABLISHED. KUBE-FORWARD (which sits above CACHE_ENFORCER) accepts ESTABLISHED with a terminating -j ACCEPT, so the ClientHello never reaches CACHE_ENFORCER. The string rule can only ever see the SYN (no payload) → it never matches.

enforce-cache.sh inserts at FORWARD position 1 (ahead of KUBE-FORWARD), where it does work — but it's a one-shot initContainer with no reconcile. kube-proxy's periodic resync (default 30s) re-inserts its own jumps at the top of FORWARD, demoting CACHE_ENFORCER below the ESTABLISHED-accept. After the first resync, enforcement is silently dead for the life of the node.

Data point: same versions, opposite outcome, on main

Two back-to-back torchtitan-test jobs on main, byte-identical resolves:

	job 80953221424 (FAIL)	job 80955014467 (OK)
pip start	03:34:12 UTC	03:53:06 UTC
torchao / torchcomms / torch / spmd-types	0.18.0.dev20260612 / 0.3.0.dev20260611 / 2.13.0.dev20260610 / 0.2.1	identical
spmd-types fetch	files.pythonhosted.org → RST ×5 → fail	downloaded fine

The earlier job failed and the later one passed — opposite of "index was broken then fixed", and consistent with the node aging past its first kube-proxy resync (enforcer effective at init, neutered ~30s later). Outcome is decided by node iptables state at run time, not by code or versions.

[jeanschmidt]

After discussion we agreed that pypi-cache is not serving its original goal, as we abandoned the objective of modernizing the CI with enforcement of best practices and instead decided to do a simple "drop replacement" of the infra with a more modern architecture.

So, pypi-cache, today, is serving little to no benefit, and given our goals for the future, it does not make sense to maintain it.

The plan now is to remove pypi-cache and cache-enforcer in one of the regions as an experiment and let it soak for a while. Especially wait for one or two waves of build jobs caused by some changes that force docker builds. If we observe stability the goal is to remove it completely and reduce our maintenance surface.

[huydhn]

Here is the PR to remove pypi-cache and cache-enforcer on ue1 #749 together with the deployment plan. My proposal is to drain the traffics from ue1 outside working hours, deploy the change plus cleaning up, then resume the traffics