Skip to content

fix(warm-replay): replay a KB recipe into a pip-installed framework - #1563

Open
BaoYunkai wants to merge 2 commits into
mainfrom
bugfix/yunkai/warm-replay-nogit-required-timeline
Open

BaoYunkai wants to merge 2 commits into
mainfrom
bugfix/yunkai/warm-replay-nogit-required-timeline

Conversation

@BaoYunkai

Copy link
Copy Markdown
Collaborator

Problem

A Recipe whose patch timeline is non-empty replays as required, and two guards
on that path refuse a tree a pip-installed framework always presents.

  1. _apply_warm_patches refused any required timeline whose tree had no git
    HEAD. A framework installed from a wheel never has one, so every code-level
    optimization a session published became unreplayable the moment it reached
    the KB: warm replay failed with missing_git_head before booting a server.

  2. An overlay the tree already carries is applied as a satisfied no-op, writing
    nothing and recording neither a snapshot manifest nor nogit backups.
    Promotion and rollback both read those absent artifacts as "cannot unwind
    this tree", so a replay that had already reproduced its gain was rejected and
    then failed with stop_reason=warm_replay_rollback_failed.
    _rollback_combined_warm also only ever restored from a git snapshot
    manifest, so a nogit tree it had written to could not be unwound either.

Fix

What promotion needs is a way to unwind the tree if the replay is rejected, not
a sha. A git checkout answers with a pre_sha and a snapshot manifest; a
pip-installed framework answers with the backups nogit wrote as it applied. The
missing_git_head refusal is removed and promotion accepts either channel.

Absent artifacts cannot carry "nothing was written" on their own -- a no-op
apply and an apply whose artifacts were lost record the same nothing, and only
the first is safe to leave standing. Each tree now states whether the round
wrote to it. A no-op tree promotes (promotion is reached only once every
required overlay applied, so it already holds what is being promoted) and is
skipped by the rollback; a tree that was written to still has to answer with a
channel. A record predating the flag is read as written-to, so a resume that
lost its manifest fails exactly as before.

_rollback_combined_warm now routes through _revert_warm_patch_state, which
picks the channel the apply recorded.

Test plan

  • test_warm_patch_apply.py, test_warm_replay.py (101 passed)
  • orchestrator/actions/executors/tests, orchestrator/tests,
    test_coordinator_async_batch2_unit.py (690 passed, 1 skipped; 2 failures
    also present on unmodified main)
  • ruff check . + ruff format --check .
  • Pod: MiniMax-M3-MXFP4 vLLM session reproduced +23.88% from the KB recipe
    and closed normally (stop_reason=sweep_done, crash_count=0)

BaoYunkai added 2 commits September 17, 2026 14:52
… tree

A Recipe whose patch timeline is non-empty is replayed as required, and
_apply_warm_patches refused any required timeline whose tree had no git HEAD.
A pip-installed framework never has one, so every code-level optimization a
session published became unreplayable the moment it reached the KB: warm
replay failed on missing_git_head before booting a server, and the recipe's
measured gain could only ever be re-earned from scratch.

The guard stood in for what promotion actually needs, which is a way to unwind
the tree if the replay is rejected, not a sha. Nogit already supplies that --
it backs up every file it patches, and _revert_warm_patch_state has since
learned to restore from those backups -- so the refusal outlived the gap it
covered. Promotion now accepts either channel: a git checkout's pre_sha plus
snapshot manifest, or the backups a nogit apply recorded. A tree offering
neither is still refused, since nothing could unwind it.

The two tests that asserted the refusal now assert the replay, and a third
covers the revert that a rejected replay depends on.

Suites: test_warm_patch_apply.py, test_warm_replay.py (139 passed);
orchestrator/actions/executors/tests + orchestrator/tests (483 passed,
1 skipped); ruff check + ruff format --check clean.
An overlay a required timeline replays can already be present in the tree --
a framework shipped carrying it, or an operator applied it to boot the server
the replay needs. The nogit applier recognises that and returns a satisfied
no-op, writing nothing and recording no backups. Promotion and rollback both
read those absent artifacts as "cannot unwind this tree" and failed the
replay: the recipe reproduced its gain, promotion refused the checkout, and
the rollback that followed set stop_reason=warm_replay_rollback_failed.

Absent artifacts cannot carry that meaning on their own -- a no-op apply and
an apply whose artifacts were lost record the same nothing, and only the
first is safe to leave standing. Each tree now states whether the round wrote
to it. A no-op tree promotes (promotion is reached only once every required
overlay applied, so it already holds what is being promoted) and is skipped by
the rollback. A tree that was written to still has to answer with a channel,
and a record predating the flag is read as written-to, so a resume that lost
its manifest fails exactly as before.

_rollback_combined_warm also only ever restored from a git snapshot manifest,
so a nogit tree it *had* written to could not be unwound either. It now routes
through _revert_warm_patch_state, which picks the channel the apply recorded.

Suites: test_warm_replay.py (101 passed); + executors/tests, orchestrator/tests,
test_coordinator_async_batch2_unit.py (690 passed, 1 skipped, 2 pre-existing
failures also present on the unmodified branch); ruff check + format clean.
@BaoYunkai
BaoYunkai requested a review from a team as a code owner September 18, 2026 04:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant