Skip to content

Preserve accepted upstream Python source through measurement and replay - #477

Open
iraj465 wants to merge 22 commits into
mainfrom
fix/accepted-source-materialization
Open

iraj465 wants to merge 22 commits into
mainfrom
fix/accepted-source-materialization

Conversation

@iraj465

@iraj465 iraj465 commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

GEAK can receive Hyperloom's latest accepted source snapshots and still import an installed regular package. Sparse snapshots also cannot represent a deleted module. The resulting benchmark can measure a different baseline from the one in the handoff. This change consumes a versioned, complete source bundle and requires proof for the selected measurements and replay before returning an eligible source-bearing result.

The interface validates accepted layer coverage, complete file hashes/modes, import ownership and deletions, stages immutable source and canonical helpers into the evaluation directory, and preserves the import order: verification bootstrap, GEAK-authored overlay, accepted upstream source, backend defaults, inherited paths. Publication supports NFS through an exclusive destination claim and manifest-last publication; concurrent incomplete claims and interrupted copies are refused without replacement.

The serving Python processes provide identity-bound observations around the timed hot rounds. Measurement records bind the exact launch capsule, source request, overlay inventory, raw measurements and summary. Persisted acceptance additionally requires confirmed teardown. Normalization verifies the selected Setup and fresh Director measurements, then stages an immutable replay wrapper. Missing proof remains diagnostic and cannot become an accepted gain or a permanent knowledge export. Per-kernel source attribution is explicitly unavailable until each operation has its own measurement provenance.

Version 1 supports regular pure-Python packages with unambiguous import ownership. Unsupported compiled artifacts, namespace/package deletion, unresolved source coverage and persistent unobserved workers are refused. The runtime covers observed persistent processes and guarded Python imports; native process creation bypassing Python auditing and arbitrary dynamic execution are outside that contract. A source seal does not certify GPU correctness, model/workload identity or native accuracy evaluation. Handoffs without source changes retain their existing behavior.

Refs #475. Paired producer: AMD-AGI/Hyperloom#1562 constructs the versioned descriptor. This is separate from PR452's launch-setting transport and PR476's optional post-measure callback. It does not change the completed online A/B or establish PR452-specific performance gains.

Validation:

  • Complete frozen L0 at 3903818c: 2,498 tests and 90 subtests passed, 5 skipped; 97.05% coverage clears the unchanged 97% gate. Python 3.11.16 / Node 20.20.2; all blocking workflow steps passed. Ruff retains the same 1,054 existing advisory findings as base, with no findings in the new Python files.
  • Independent final staging suite: 52 tests pass on ext4 and actual NFS. Actual subprocess interruption, concurrency, completed-copy reuse and replay import after original-source removal pass.
  • Independent runtime and measurement review includes real process/cleanup checks and negative controls for stale bytecode, added helpers, symlink dependencies, escaped workers and modified replay content. The original false-positive reproductions are retained.
  • Actual Hyperloom producer recognizes the consumer capability; direct producer/consumer materialization and relocation probes pass. GPU serving and a new matched online A/B remain to be qualified.

The first full run is retained as failed: an existing recovery test leaked ignored SIGTERM into later tests, and a renamed-interpreter fixture could not locate the Python 3.11 standard library. Test-only corrections restore the original signal disposition and verify the renamed worker actually starts. Controlled reproductions and the corrected ordered 169-test suite passed before the final full run; runtime cleanup behavior and assertions were unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant