[RFC]: afd-plugin project roadmap

[hsliuustc0106]

Summary

AFD is an external vLLM plugin for Attention–FFN Disaggregation.

The roadmap has six workstreams:

1. compatibility, CI/CD, and release foundation;
2. DeepSeek-V4 and GLM-5.2 model expansion;
3. prefill workloads and chunked-prefill/prefix-cache compatibility;
4. NVIDIA and AMD large-scale serving recipes;
5. Kimi-K3 prefill feasibility;
6. repository-backed agent skills.

These are workstreams, not release phases. Dependencies determine when a support claim is ready, while independent work may proceed in parallel.

Goals

• Keep AFD aligned with a deliberate, maintainable vLLM compatibility window.
• Make support claims precise and backed by model-, backend-, and topology-specific evidence.
• Provide reproducible NVIDIA and AMD large-scale serving recipes.
• Expand model support without silently coupling all backend lanes.
• Validate realistic long-prefill and agentic workloads, including chunked prefill and prefix caching.
• Establish basic every-PR CI and tag-triggered GitHub/PyPI releases.
• Provide thin agent skills that make repository-owned workflows easier to run and review.

Non-goals

• Reframe AFD as a vLLM-Ascend plugin.
• Create release phases or fixed delivery deadlines.
• Require every model to be validated on every backend before any support claim can be made.
• Block core plugin releases on unavailable rack-scale hardware or external collaboration.
• Claim performance or production readiness from launch-only evidence.
• Duplicate deterministic implementation logic inside agent skill prose.

Plugin boundary and layered support claims

Support must be stated in layers rather than as one global yes/no claim.

Layer	Claim	Minimum evidence
1. Core vLLM compatibility	The plugin loads, configures, isolates non-AFD behavior, and satisfies compatibility contracts for one exact vLLM version.	CPU-safe contract tests, package build/install, plugin-disabled isolation, and pinned-version checks.
2. Model/backend validation	A named model works correctly on a named backend.	Model adaptation review, correctness/accuracy evidence, exact software environment, and backend-specific E2E results.
3. Recipe/topology validation	A named topology has a reproducible and measured serving recipe.	Exact hardware/network topology, launch commands, correctness, performance results, sustained-load evidence, and named validation review.

A model may therefore be validated on one backend without waiting for all other backends. Missing evidence must be labeled experimental or unverified rather than generalized into a broad support claim.

Dependency structure

The compatibility and delivery foundation is a prerequisite for durable support claims. After a target vLLM version is selected, model expansion and prefill feature adaptation can proceed independently. Their evidence feeds large-scale recipes and the Kimi-K3 feasibility study. Agent skills follow stable repository workflows and may be added alongside them.

This ordering is about dependencies and evidence, not release phases.

Workstream 1: compatibility, CI/CD, and releases

vLLM alignment policy

• Review vLLM alignment at least monthly.
• In steady state, target every second stable vLLM minor line, initially the even-numbered lines (0.26, 0.28, and so on).
• Treat this as a two-release cadence, not a claim that even-numbered releases are intrinsically safer.
• Allow the supported AFD line to remain at most two stable minor release lines behind upstream after the initial bootstrap upgrade.
• Permit an earlier exceptional alignment for a critical fix, security need, or required model capability.
• Record each alignment decision even when the decision is to stay on the current target.

At the time of this RFC, the latest stable vLLM release is 0.25.1, while AFD targets 0.19.1. The first prospective even-line target is therefore 0.26.x once that line is stable and selected through the monthly review.

Mirrored versioning

AFD package versions communicate exact vLLM compatibility:

• AFD 0.26.0 targets vLLM 0.26.0.
• AFD 0.26.0.post1 is an AFD-only hotfix for that same vLLM version.
• AFD 0.26.1 is reserved for compatibility with vLLM 0.26.1.

Initial CI and CD

Every PR should run accessible CPU-only gates:

• lint and formatting checks;
• CPU-safe unit and contract tests;
• package build and installation checks;
• plugin-disabled isolation checks;
• compatibility checks against the pinned vLLM contract.

Hardware fleets are not assumed to be available for every PR. Hardware evidence is reviewed in the implementation PR and retained in the corresponding recipe.

When a release is declared ready, a matching version tag should trigger CD. Initially, the release gate is CPU-only and publishes both a GitHub Release and PyPI artifacts. The first planned hardware CI extension is a GPU E2E test using DeepSeek-V2-Lite when suitable devices become available.

Workstream 2: model expansion

The near-term committed model targets are:

• DeepSeek-V4;
• GLM-5.2.

Adaptation follows the native vLLM lifecycle and the plugin boundary. Each model receives independent backend validation; validation on one backend does not imply validation on NVIDIA, AMD, and Ascend simultaneously.

A model adaptation is complete only when its implementation PR and corresponding recipe identify the exact vLLM version, backend, environment, correctness method, tested modes, and known exclusions.

Workstream 3: prefill workloads and cache/scheduler compatibility

This is a standalone workstream because its compatibility matrix and benchmark evidence are reused by model and hardware recipes.

Initial scope

• Focus on the prefill node only.
• Do not claim decode-side latency improvements or mixed prefill/decode scheduling benefits in this initial scope.
• Establish correctness, stability, and reproducibility before interpreting performance.

Step-by-step compatibility matrix

Validate these configurations independently and in order:

1. chunked prefill off, prefix caching off — baseline;
2. chunked prefill on, prefix caching off;
3. chunked prefill off, prefix caching on;
4. chunked prefill on, prefix caching on.

Functional compatibility does not itself claim a speedup. Results must report benefits, regressions, and unsupported combinations.

Workloads

Use both of the following:

1. fixed long-prefill inputs at 16K, 32K, and 128K tokens;
2. multi-turn coding-agent trace replay with growing shared prefixes and incremental tool-output/context deltas.

Use the upstream vLLM benchmark harnesses and test cases from the exact vLLM tag targeted by the mirrored AFD release. AFD should retain only thin launch/configuration wrappers and evidence records; it should not fork the benchmark implementation. The agreed lengths and replay trace are inputs to the upstream harness.

Primary reporting should include correctness, failure/hang behavior, TTFT including tail behavior, SLO-constrained prefill goodput, and prefix-cache effectiveness where applicable. Exact SLOs and benchmark parameters belong in the child study or recipe.

Workstream 4: NVIDIA, AMD, MUSA large-scale serving recipes

NVIDIA

A validated vLLM-native AFD recipe for an NVIDIA GB200 NVL72-class deployment is a good-to-have roadmap outcome. FastAFD is a useful technical reference and a potential collaboration path, but it is a separate serving system rather than the host framework for this plugin.

This outcome needs OSS participation and access to suitable rack-scale hardware. It is strategically required but does not block core releases until a consenting owner, collaboration path, and hardware capacity are available.

AMD

• Target the topology capabilities required by AFD: a rack-/pod-scale high-bandwidth scale-up domain, likely in a SuperPod-class environment. Every published recipe must still disclose the exact tested accelerators, network/fabric, node count, topology, and software stack.

MUSA @shanyulu

• evaluating an experimental MUSA backend for AFD, initially targeting synchronous decode, eager correctness, and a reproducible DeepSeek recipe

Recipe evidence levels

An experimental recipe must include a reproducible environment and launch, successful serving, correctness against a colocated vLLM baseline, and exact system disclosure.

A validated recipe is preferred and additionally includes scale curves, throughput and latency, multiple Attention:FFN ratios, sustained load, and failure/recovery behavior. No fixed speedup threshold is assumed before measurement, but launch-only evidence is insufficient.

Workstream 5: Kimi-K3 prefill feasibility

Kimi-K3 should begin as a feasibility study rather than a committed implementation. Its heterogeneous attention structure makes the AFD performance boundary uncertain.

Initial analysis should focus on prefill and use SLO-constrained prefill goodput, including tail TTFT as a hard constraint. Compare AFD or PD+AFD with:

• a tuned colocated vLLM baseline;
• native prefill/decode disaggregation where applicable.

Implementation graduates to a support target only if at least one documented workload and topology shows a defensible benefit. A well-supported negative result is an acceptable outcome.

Workstream 6: repository-backed agent skills

Initial maintainer/contributor skills should cover:

1. aligning to the next targeted vLLM version;
2. adapting and validating a model;
3. creating and reviewing a hardware recipe;
4. running E2E tests;
5. preparing a mirrored-version release.

Skills must be thin orchestrators over repository-owned scripts, templates, tests, and checklists. Deterministic behavior belongs in the repository and CI, not duplicated in skill prose. Skills handle discovery, required inputs, sequencing, and reporting.

Evidence and review policy

Hardware-backed claims are reviewed in both places:

• the implementation PR, where reviewers evaluate the code and evidence;
• the corresponding recipe, which retains the reproducible environment, topology, commands, correctness method, results, limitations, and validator.

Risks and mitigations

Risk	Mitigation
Monthly alignment creates continuous churn.	Upgrade only on the selected two-release cadence; use the monthly review to record stay/upgrade decisions.
The current upstream gap makes the first alignment unusually large.	Treat it as bootstrap work and require contract, model, package, and isolation evidence before claiming the new line.
Layered claims become confusing.	Publish exact version/model/backend/topology status and never infer an untested layer.
Large-scale hardware is unavailable.	Keep recipes strategically required but non-blocking and visibly Help Wanted until resources exist.
FastAFD behavior is copied without preserving the vLLM boundary.	Use it as a reference/collaboration source; implement a vLLM-native AFD plugin recipe.
Chunked prefill and prefix caching interact unexpectedly with AFD scheduling.	Validate baseline, each feature independently, and the combined mode using the matching upstream benchmark revision.
Agentic benchmarks become synthetic and unrepresentative.	Retain fixed-length isolation cases and add replayable multi-turn coding-agent traces.
Kimi-K3 offers no practical prefill gain.	Treat negative feasibility evidence as a valid result rather than forcing implementation.
Agent skills drift from actual workflows.	Keep scripts, templates, checklists, tests, and CI authoritative; skills only orchestrate them.

Feedback requested

Discussion should focus on:

• the monthly review and two-release vLLM alignment cadence;
• evidence required for each layer of support;
• prioritization and backend validation for DeepSeek-V4 and GLM-5.2;
• the upstream benchmark coverage and four-mode prefill compatibility matrix;
• access, ownership, and collaboration for NVIDIA GB200 NVL72 and AMD scale-up recipes;
• the benefit boundary for Kimi-K3 heterogeneous-attention prefill;
• the repository workflows that should exist before each thin agent skill is added.

References

• Prior roadmap discussion: [RFC]: AFD project roadmap and release gates #153
• vLLM 0.25.1 release
• vLLM 0.25.1 supported-model matrix
• vLLM 0.25.1 benchmarks
• Serving agentic workloads with vLLM and Mooncake
• Kimi-K3 preview in vLLM
• FastAFD repository

CC

@jiangkuaixue123 @specture724 @jiaran-king @yujuancao07 @ShwStone

[jiangkuaixue123]

Thank you for sharing the roadmap. I'll take some time to digest it first, and we can have a more detailed discussion next week.

[david6666666]

I think there should be a sub-RFC for ModelRunner v2.

[shanyulu]

The current roadmap covers NVIDIA, AMD, and existing Ascend paths,
but does not mention Moore Threads MUSA.

We are evaluating an experimental MUSA backend for AFD, initially
targeting synchronous decode, eager correctness, and a reproducible
DeepSeek recipe.

Would maintainers prefer a separate RFC issue or a design-only draft PR?

[hsliuustc0106]

The current roadmap covers NVIDIA, AMD, and existing Ascend paths, but does not mention Moore Threads MUSA.

We are evaluating an experimental MUSA backend for AFD, initially targeting synchronous decode, eager correctness, and a reproducible DeepSeek recipe.

Would maintainers prefer a separate RFC issue or a design-only draft PR?

could you add a subissue for MUSA backend? I will add it as an option in this roadmap

[chopper0126]

I have some thoughts on the roadmap.

1. The performance benefits of using the asynchronous AFD prefill with CAMAsyncAFDConnector were verified on the DeepSeek V3.2 model with fewer layers. These benefits can be further verified on the complete model. The same applies to EPLB.
2. In AFD prefill, the ffn can be separately fed into the graph, saving the kernel delivery time and significantly improving performance.
3. CAMAsyncAFDConnector for asynchronous AFD prefill can be verified on the 950PR. After the 950DT is released in Q4, heterogeneous deployment can be verified, and it is expected that even greater benefits will be achieved.(To adapt this operator to the 950PR, you can use the operator generation agent at https://gitcode.com/cann/cannbot-skills to migrate the 910C to the 950.)
   @jiangkuaixue123 @hsliuustc0106

[shanyulu]

Added the requested MUSA backend sub-issue here: #173

[hsliuustc0106]

1. The performance benefits of using the asynchronous AFD prefill with CAMAsyncAFDConnector were verified on the DeepSeek V3.2 model with fewer layers. These benefits can be further verified on the complete model. The same applies to EPLB.

• full model evaluation, I think we can work on DS-V4-flash @jiangkuaixue123 @ShwStone @specture724
• EPLB benefits with more expert redundancy

2. In AFD prefill, the ffn can be separately fed into the graph, saving the kernel delivery time and significantly improving performance.

do we have a profiling about current kernel launch time or have we already taken the cudagraph? @jiangkuaixue123 @yujuancao07

3. CAMAsyncAFDConnector for asynchronous AFD prefill can be verified on the 950PR. After the 950DT is released in Q4, heterogeneous deployment can be verified, and it is expected that even greater benefits will be achieved.(To adapt this operator to the 950PR, you can use the operator generation agent at https://gitcode.com/cann/cannbot-skills to migrate the 910C to the 950.)> @jiangkuaixue123 @hsliuustc0106

This can be left for Q4