feat(eval): coordinator scaling evaluation with race-safe live dispatch - #96
Open
drunkcoding wants to merge 7 commits into
Open
feat(eval): coordinator scaling evaluation with race-safe live dispatch#96drunkcoding wants to merge 7 commits into
drunkcoding wants to merge 7 commits into
Conversation
Replace the hardcoded version = "0.0.1" with a git-tag-driven version. - pyproject.toml: add setuptools-scm>=8 to build requires, make version dynamic, configure [tool.setuptools_scm] with a fallback_version for non-git builds. - morphling/__init__.py: expose __version__ from the generated _version.py, falling back to importlib.metadata. - Dockerfile: the build context excludes .git and installs with --no-build-isolation, so install setuptools-scm and inject the version via SETUPTOOLS_SCM_PRETEND_VERSION_FOR_MORPHLING (empty => fallback_version). - gitignore the generated morphling/_version.py. Verified in-container: docker build --build-arg MORPHLING_VERSION=0.1.0 yields morphling.__version__ == 0.1.0 with _C/_Msg/_GreenCtx importing.
Add single- and multi-coordinator evaluation over the real dispatch path and fix the concurrency defects that blocked it. Runtime (csrc/backend, morphling/hooks/autograd.py): - DispatchMatMulAsync returns a distinct operation id and callers wait on it, so concurrent GEMMs no longer alias each other's output buffers. - Guard the result hand-off with a mutex and condition variable; WaitMatMul clones the completed tensor and rejects out-of-range operation ids. - Add PartitionTracker::ClaimIdlePartitions and ReassignPartitionToDevice so concurrent SendIdlePartitions callbacks cannot double-send a shard. - Bound the operation id by kMaxLifetimeOperationCount and fail fast instead of writing out of bounds; materialize contiguous forward operands. Evaluation (morphling/runtime, scripts): - Sampled coordinator CPU, RSS, NIC, and phase metrics. - Single-coordinator device-count sweep and one/two-coordinator strong and weak scaling with per-iteration breakdowns. - Three-panel coordinator scaling figure pipeline. Tests: C++ operation-id and partition-claim concurrency tests; Python coordinator metrics, scaling, workload, dispatch-safety, and figure tests. Docs: document the live-dispatch synchronization invariants and the operation-id bound in docs/GEMM_ID_ISSUES.md.
- Apply ruff-format (0.9.4) and clang-format to the coordinator sources and tests so the Formatting pre-commit job passes. - Import assert_never compatibly on Python 3.10 (fall back to typing_extensions) in the multi-coordinator scaling modules; typing.assert_never is 3.11+ and the project supports 3.10. - Add tests/python/unit/conftest.py to skip coordinator-evaluation tests when psutil is absent, so the CPU smoke job (pytest/torch/numpy only) no longer aborts collection on module-scope heavy-dependency imports. Full/Docker environments still collect and run them.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds single- and multi-coordinator evaluation over the real dispatch path and fixes the concurrency defects that previously blocked live optimizer-in-the-loop training.
Runtime fixes (
csrc/backend,morphling/hooks/autograd.py)DispatchMatMulAsyncnow returns a distinct operation id and callers wait on it, so concurrent GEMMs no longer alias each other's output buffers.WaitMatMulclones the completed tensor and rejects out-of-range operation ids.PartitionTracker::ClaimIdlePartitions/ReassignPartitionToDevicemake the IDLE→RUNNING claim atomic, so concurrentSendIdlePartitionscallbacks cannot double-send a shard (root cause of a nondeterministic NaN/hang).kMaxLifetimeOperationCountwith fail-fast behavior instead of out-of-bounds writes; forward operands are materialized contiguous to match the row-major backend.Evaluation harness (
morphling/runtime,scripts)Docs: live-dispatch synchronization invariants and the operation-id bound documented in
docs/GEMM_ID_ISSUES.md.Test plan
test_operation_id(3) andtest_partition_tracker_claim(4) pass in Docker.Scope / limitations
Measured evaluation is OPT-125M, one/two coordinators on a single host with CPU affinity + first-touch memory and Gloo over loopback (a lower bound; no physical NIC-to-NIC routing). It does not claim OPT-6.7B, 4/8 coordinators, multi-host, or physical-NIC behavior. The paper frames the 8,192-device point as calibrated emulation plus analytical projection, not a multi-coordinator run.