fix(upload): time-consistent replay so delegated sessions don't strand at status=running - #94
Open
Diego Colombo (colombod) wants to merge 3 commits into
Open
fix(upload): time-consistent replay so delegated sessions don't strand at status=running#94Diego Colombo (colombod) wants to merge 3 commits into
Diego Colombo (colombod) wants to merge 3 commits into
Conversation
…sions run_upload previously fed events whole-session, parent-first, with no ordering relative to the server's concurrent per-session drain -- so a delegated sub-session (or its parent) could be left status=running when a cross-session reopen was processed after that node's own session:end. Feed events in true global timestamp order across the session-tree closure (lazy heapq.merge over per-line generators -- O(sessions) memory, never loads a file whole), and pace by the events' own inter-event gaps capped at max_gap_s (default 2.0s) so a spawned sub-session drains before its parent resumes. No server change and no /status/drain signal -- ordering is derived purely from the recorded timestamps. Reuses the existing parse/build_payload/POST/retry/auth path verbatim; only the feed order + pacing + tracker triggers changed. Validated on an isolated server: child+parent completed 20/20; non-delegated upload unaffected; peak RSS flat (~41MB) uploading a 74.9MB single-session file; zero /status calls. 564 existing + 10 new tests green. Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
…sion)
test_project_version asserted the pyproject version equals a hard-coded literal
("0.1.3"). It catches no defect -- it only mirrors one hand-edited string
against another -- and breaks on every routine version bump and every in-flight
PR based on a pre-bump commit (as #93's 0.1.4 bump just did, leaving main red).
The useful pyproject contracts (name, requires-python, license, deps, no
amplifier.modules entry point) are kept.
Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Match the lock's editable self-package version to pyproject (bumped to 0.1.4 by #93); no dependency changes. Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Diego Colombo (colombod)
force-pushed
the
fix/upload-time-consistent-replay
branch
from
August 17, 2026 11:14
d205b7b to
f4c8dfb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a recorded session is uploaded,
run_uploadfeeds events one whole session at a time, parents before children, in strict file order, with nothing relating that feed to the server's own drain order. The server drains each session's events on an independent, concurrent queue. So for a session that used delegated sub-agents, a cross-session write that (re)opens a node — the parent'sdelegate:agent_spawnedopening the child, or the child'ssession:startopening the parent — can be processed after that node's ownsession:end. The node is then left stranded atstatus="running"in the graph even though the upload completed with no error.This is an upload-ordering artifact only: it does not occur for live, real-time capture (the events are naturally spaced), and it is invisible at upload time (every POST returns 2xx).
Fix (client-side only)
Make the uploader replay time-consistently, so a spawned sub-session drains before its parent resumes — reproducing the real capture timing that is race-free:
heapq.mergeover per-line generators. A spawned sub-session's events therefore interleave before its parent's later-timestamped resume, exactly as they occurred.max_gap_s(default 2.0s), derived purely from the recordedtimestampfields.Everything else is reused verbatim — parsing,
build_payload, the POST + bounded-retry loop, per-attempt auth, reconciliation, and progress. Only the feed order + pacing + the tracker's start/complete triggers changed.Explicitly NOT in this change
POST /events. No/statuspolling, no drain/barrier signal. (Enforced in tests: the client mock isspec=["post"], so any non-POST call raises.)What this gives user data uploads
completed, so the uploaded graph faithfully represents what happened instead of showing phantom "still running" sessions.events.jsonlis ever read whole. Measured: a 74.9 MB single-session file uploads inside a ~41 MB process; 430× more events / ~1960× more bytes cost +944 KB peak RSS.accepted == written, 0 dead-letter/residual).Evidence
Validated against an isolated server instance with small real sessions:
completed20/20 trialsevents.jsonl; RSS independent of event count/statusPOST /eventsScope / not claimed
max_gap_s, never a wrong result.IncompleteSessionco-labelling (its own PR).Test plan
uv run pytest(module) — green (574).uv run ruff check/pyright— clean.tests/test_replay_ordering.py: global timestamp ordering, parent/child interleave (childsession:endbefore parent's; parent spawn before childsession:end), capped-gap pacing withevent_delay_sfloor, and a mechanical assertion that no non-/eventsrequest is ever made.