网络后端替换

[gongxinyue]

请问若想要替换网络后端，主要需要修改的接口有哪些？这里是否类似Astra-sim中的自定义网络后端？
https://astra-sim.github.io/astra-sim-docs/network-backend/custom-network-backend.html

[tianhao909]

【Manual reply】
应该有基于Astra-sim修改的接口 （在 Astra-sim 抽象基础上做了扩展）； 具体可以参考agent抓的调用栈接口等

[tianhao909]

[Reply from Agent A] [For reference only]

CN

你好！

是的，这里的机制完全类似于 Astra-sim 中的自定义网络后端机制。SimAI 的 astra-sim-alibabacloud 组件基本保留了 Astra-sim 对于下层网络的抽象方式，所以官方文档中的适配思路依然完全适用。

如果想要替换网络后端并接入你自己的平台，你主要需要完成以下几处代码的编写和修改：

1. 继承并实现自定义网络接口：
   你需要实现一个继承自 AstraSim::AstraNetworkAPI 的子类（位于 astra-sim-alibabacloud/astra-sim/system/AstraNetworkAPI.hh），主要需重写以下交互接口和时间调度接口：

   • sim_send(...)：处理如何将消息派发至你的网络模拟器并在完成时触发回调。
   • sim_recv(...)：处理数据接收事件的注册。
   • sim_schedule(...)：实现底层的延迟/事件调度器。
   • sim_get_time()：返回当前网络模拟器的全局时间。
   • 以及 sim_init, sim_finish 等声明周期控制函数。

2. 在入口函数（main）中完成注入：
   你可以参考目前的 NS3 后端实现（位于 astra-sim-alibabacloud/astra-sim/network_frontend/ns3/AstraSimNetwork.cc 的 main 函数）。

   • 将原代码中的 networks[j] = new ASTRASimNetwork(j ,0); 替换为你自己实现的类 new CustomNetworkAPI(j);。
   • 保证构造好的 networks[j] 被传装进 AstraSim::Sys 中完成初始化。

最后再适配一下 scripts/build.sh 或 CMake 将你依赖的网络后端编入可执行程序即可。欢迎引入新的网络后端并在社区继续交流尝试情况！

====

EN

Hello!

Yes, the mechanism here is absolutely identical to the custom network backend in standard Astra-sim. SimAI's astra-sim-alibabacloud module retains the core network abstraction from Astra-sim, meaning the official guide you linked still perfectly applies.

If you want to plug in your own custom network backend, here are the primary interfaces and code blocks you roughly need to modify:

1. Implement your Custom Network Interface:
   You need to create a class derived from AstraSim::AstraNetworkAPI (located in astra-sim-alibabacloud/astra-sim/system/AstraNetworkAPI.hh). The most critical virtual functions you need to override are:

   • sim_send(...): Handle how the message is dispatched to your custom backend and how the system is notified via callbacks when finished.
   • sim_recv(...): Register receive events.
   • sim_schedule(...): Handle internal simulation delays and timers.
   • sim_get_time(): Return the global time of your network backend.
   • Setup routines like sim_init and sim_finish.

2. Inject the Instance via the Main Entry:
   You can take the current NS-3 integration driver (astra-sim-alibabacloud/astra-sim/network_frontend/ns3/AstraSimNetwork.cc inside its main loop) as a typical example.

   • Simply find where the network is instantiated (e.g., networks[j] = new ASTRASimNetwork(j, 0);) and replace it with your newly developed wrapper like networks[j] = new CustomNetworkAPI(j);.
   • Ensure these customized objects are then passed into AstraSim::Sys setup.

After that, you'll just need to adjust your scripts/build.sh or compile configurations to link against your external simulator. Feel free to try it out and let us know if you run into any barriers!

[tianhao909]

[Reply from Agent B] [For reference only]

CN

网络后端替换分析（Issue #261）

1. 结论先说

可以，SimAI 当前的网络后端替换思路整体上和 Astra-sim 的 custom network backend 是类似的，核心抽象入口就是：

• astra-sim-alibabacloud/astra-sim/system/AstraNetworkAPI.hh

如果要替换网络后端，最主要需要修改/新增的是一个 AstraNetworkAPI 的具体实现，而不是去大改 collective 算法层或 workload 生成层。

---

2. 主要需要关注/修改的接口

核心接口

需要重点实现以下接口：

• sim_init(AstraMemoryAPI* MEM)
• sim_get_time()
• sim_schedule(timespec_t delta, void (*fun_ptr)(void*), void* fun_arg)
• sim_send(...)
• sim_recv(...)
• sim_finish()
• sim_comm_size(...)

这些接口都定义在：

• astra-sim-alibabacloud/astra-sim/system/AstraNetworkAPI.hh

调用链相关接口

上层 collective 并不会直接调用 ns-3，而是通过 Sys 层中转，主要入口包括：

• front_end_sim_send(...)
• front_end_sim_recv(...)
• sim_send(...)
• sim_recv(...)

定义位置：

• astra-sim-alibabacloud/astra-sim/system/Sys.hh

现有参考实现

可以参考已有几个 backend：

• astra-sim-alibabacloud/astra-sim/network_frontend/analytical/AnalyticalNetwork.*
• astra-sim-alibabacloud/astra-sim/network_frontend/phynet/SimAiPhyNetwork.*
• astra-sim-alibabacloud/astra-sim/network_frontend/ns3/AstraSimNetwork.h

如果只是想“新增一个后端”，最小改动通常是在 network_frontend/ 下新增一个目录和实现类。

---

3. 是否类似 Astra-sim 中的自定义网络后端？

是，思路类似，但不是完全照搬。

相似点

• 都是通过统一抽象接口解耦 collective/system 层与底层网络模拟器
• 都需要提供 send / recv / schedule / time 等能力
• 都支持把网络 simulator 当作可插拔组件

不同点

SimAI 在 Astra-sim 抽象基础上做了扩展，尤其是：

• sim_request 中增加了 vnet、layerNum、flowTag 等字段
• 系统层 Sys 增加了自己的前端调度逻辑
• 已有 analytical / physical / fast-backend 等多种后端形态

因此，如果要在 SimAI 中替换网络后端，建议参考 Astra-sim 的设计思路，但具体实现要以 SimAI 当前 AstraNetworkAPI + Sys 的调用链为准。

---

4. 最小可行修改方案

建议采用下面的最小方案：

方案：新增一个 backend 类

1. 在 astra-sim-alibabacloud/astra-sim/network_frontend/ 下新建目录，例如 mybackend/
2. 新增一个类继承 AstraNetworkAPI
3. 实现以下最小接口：
   • sim_init
   • sim_get_time
   • sim_schedule
   • sim_send
   • sim_recv
   • sim_finish
4. 在 backend 初始化/构建入口增加一个分支，实例化这个新类
5. 尽量不改 Sys、collective、AICB

为什么这是最小改动

因为 collective 层当前已经只依赖统一抽象接口，并不直接依赖具体网络后端。
所以只要新 backend 满足 AstraNetworkAPI 协议，理论上上层逻辑不需要大改。

---

5. 测试前置条件

是否依赖额外数据？

最小功能验证通常不依赖额外数据。

可以直接用仓库已有：

• example workload
• topo 生成脚本
• SimAI.conf

来验证：

• backend 初始化是否成功
• send/recv/callback 是否闭环
• 小规模 collective 是否能跑通

什么情况下需要额外数据？

如果你希望验证：

• 与真实网络精度一致
• 与 ns-3 结果一致
• 拥塞控制/路由/拓扑细节

则通常需要额外：

• trace 数据
• latency/bandwidth profile
• backend 专属配置文件
• 或真实/半真实环境

---

6. README 是否需要更新？

建议更新。

原因是：

• 代码里已经存在网络后端抽象
• 但当前文档没有明确告诉开发者“如何新增/替换 backend”
• 本 issue 本身就说明这部分使用说明不足

建议补充内容

建议在 README 或 docs 中补一节：

• Custom Network Backend
• 说明 AstraNetworkAPI 是主入口
• 列出必须实现的接口
• 说明 sim_request 字段含义
• 给出一个最小 backend 示例
• 说明如何在构建/运行入口选择 backend

---

7. 当前信息不足之处与假设

信息不足

1. 当前仓库里 backend 实例化入口没有在这轮分析中完全展开
2. ns3 frontend 目前看起来像示例/半成品，是否是主生产路径需要进一步确认
3. issue 没有评论，无法结合维护者回复判断官方推荐接入方式

当前假设

1. 用户说的“替换网络后端”是指接入新的 simulator/backend，而不是修改 ns-3 内部某个协议模块
2. 希望保持现有 collective / workload 流程不变
3. 目标是最小改动接入，而不是重构网络系统

---

8. 一句话回答 issue

如果想替换 SimAI 的网络后端，最主要要对接的是 AstraNetworkAPI 这一层，重点实现 sim_init / sim_get_time / sim_schedule / sim_send / sim_recv / sim_finish 等接口；整体思路与 Astra-sim 的 custom network backend 类似，但需要按 SimAI 现有的 Sys + sim_request + network_frontend 结构来接，而不是完全照搬 Astra-sim 文档。

EN

Network Backend Replacement Analysis (Issue #261)

1. Short answer

Yes — SimAI’s network backend replacement model is conceptually similar to Astra-sim’s custom network backend, and the main abstraction entry is:

• astra-sim-alibabacloud/astra-sim/system/AstraNetworkAPI.hh

If you want to replace the network backend, the main work is to add/modify a concrete implementation of AstraNetworkAPI, rather than changing the collective algorithm layer or the workload generation layer.

---

2. Main interfaces/modules to modify

Core interface

The key interface to implement is:

• sim_init(AstraMemoryAPI* MEM)
• sim_get_time()
• sim_schedule(timespec_t delta, void (*fun_ptr)(void*), void* fun_arg)
• sim_send(...)
• sim_recv(...)
• sim_finish()
• sim_comm_size(...)

These are defined in:

• astra-sim-alibabacloud/astra-sim/system/AstraNetworkAPI.hh

Call-chain related layer

Upper-layer collectives do not directly call ns-3. They go through the Sys layer, mainly via:

• front_end_sim_send(...)
• front_end_sim_recv(...)
• sim_send(...)
• sim_recv(...)

Defined in:

• astra-sim-alibabacloud/astra-sim/system/Sys.hh

Existing reference implementations

Useful references include:

• astra-sim-alibabacloud/astra-sim/network_frontend/analytical/AnalyticalNetwork.*
• astra-sim-alibabacloud/astra-sim/network_frontend/phynet/SimAiPhyNetwork.*
• astra-sim-alibabacloud/astra-sim/network_frontend/ns3/AstraSimNetwork.h

If the goal is simply to add a new backend, the minimum-change path is usually to add a new implementation under network_frontend/.

---

3. Is this similar to Astra-sim’s custom network backend?

Yes, in principle — but not identical in implementation details.

Similarities

• Both decouple the collective/system layer from the underlying network simulator through a unified abstraction
• Both require send / recv / scheduling / time-related interfaces
• Both support pluggable network simulators/backends

Differences

SimAI extends the abstraction with its own semantics, especially:

• sim_request includes extra fields such as vnet, layerNum, and flowTag
• Sys adds its own frontend scheduling logic
• There are multiple backend styles already present: analytical, physical, fast-backend wrapping, etc.

So if you want to replace the backend in SimAI, it is best to follow Astra-sim’s general idea, but implement against SimAI’s actual AstraNetworkAPI + Sys call chain rather than copying Astra-sim docs literally.

---

4. Minimum viable modification plan

Recommended minimum plan:

Option: add a new backend class

1. Create a new directory under astra-sim-alibabacloud/astra-sim/network_frontend/, e.g. mybackend/
2. Add a class derived from AstraNetworkAPI
3. Implement at least:
   • sim_init
   • sim_get_time
   • sim_schedule
   • sim_send
   • sim_recv
   • sim_finish
4. Add a small branch in the backend initialization/build path to instantiate the new class
5. Avoid changing Sys, collective logic, or AICB unless absolutely necessary

Why this is the minimum-change path

Because the collective layer already depends on the abstract API instead of a concrete simulator.
As long as the new backend satisfies the AstraNetworkAPI contract, upper layers should require little or no change.

---

5. Test prerequisites

Does testing require extra data?

For minimum functional validation, usually no.

You can reuse the existing repository assets:

• example workloads
• topology generation scripts
• SimAI.conf

These are enough to validate:

• backend initialization
• send/recv/callback closure
• small collective smoke tests

When is extra data required?

If you want to validate:

• realism against physical networks
• consistency with ns-3 results
• congestion-control/routing/topology fidelity

Then you will usually need:

• traces
• latency/bandwidth profiles
• backend-specific config
• or a physical / semi-physical environment

---

6. Should README be updated?

Yes, recommended.

Reason:

• the abstraction already exists in code
• but the developer-facing integration path is not clearly documented
• this issue itself indicates that the documentation is not explicit enough

Suggested documentation additions

Add a section such as:

• Custom Network Backend

And document:

• AstraNetworkAPI as the main extension point
• required virtual methods
• the meaning of sim_request fields
• callback and sim_schedule semantics
• how to select/register the backend in build/runtime
• a minimal example backend

---

7. Missing information and assumptions

Missing information

1. The exact backend instantiation path was not fully expanded in this analysis
2. The current ns3 frontend looks partially like a stub/reference; whether it is the full production path should be confirmed
3. The issue has no comments, so there is no maintainer guidance yet on the officially preferred extension path

Assumptions

1. “Replace network backend” means integrating a new simulator/backend, not merely changing an internal ns-3 protocol module
2. The goal is to keep the current collective/workload flow unchanged
3. The priority is minimum-change integration, not a redesign of the networking stack

---

8. One-sentence reply to the issue

If you want to replace SimAI’s network backend, the main abstraction to target is AstraNetworkAPI, especially sim_init / sim_get_time / sim_schedule / sim_send / sim_recv / sim_finish; the overall idea is similar to Astra-sim’s custom network backend, but the implementation should follow SimAI’s existing Sys + sim_request + network_frontend structure instead of copying Astra-sim’s documentation directly.

[tianhao909]

[Reply from Agent C] [For reference only]

CN

答复 Issue #261：替换网络后端

1. 是否类似 Astra-sim 的自定义网络后端？

是的，概念上高度一致。 SimAI 的仿真栈基于 astra-sim-alibabacloud，网络层与上层（Sys、workload）之间通过 AstraNetworkAPI 解耦。这与 Astra-sim 官方文档中「实现 Network API + 编写与所选仿真器绑定的 main/初始化」的思路一致。
请注意：具体虚函数名称与扩展字段以本仓库 astra-sim/system/AstraNetworkAPI.hh 为准，SimAI 在 sim_request、ncclFlowTag、BackendType 等方面相对上游文档可能有增量。

2. 若要替换网络后端，主要需要对接哪些接口？

在 AstraNetworkAPI 子类中实现（均为上层调用的契约）：

类别	接口	说明
生命周期	sim_init、sim_finish、sim_time_resolution	与仿真器生命周期、时间精度相关
时间与调度	sim_get_time、sim_schedule	推进仿真时间；非通信类延迟通过 sim_schedule 回调
通信	sim_send、sim_recv	异步语义：注册回调后尽快返回，在消息完成时调用 msg_handler
MPI 风格	sim_comm_size 等	按当前实现填空或实现真实语义
SimAI 扩展	get_backend_type、pass_front_end_report、get_BW_at_dimension	视后端是否影响队列/统计按需重写

入口程序：需参考 network_frontend/ns3/AstraSimNetwork.cc 或 network_frontend/analytical/AnalyticalAstra.cc：解析参数、按 rank 创建 network handler 与 Sys、workload->fire()、再进入所选网络模拟器的主事件循环。

构建：新后端需接入 astra-sim-alibabacloud 的编译与 scripts/build.sh 流程；当前 NS-3 路径会将 ns-3-alibabacloud 同步到 extern/network_backend/ns3-interface 再编译——换引擎时通常用新的 extern 目录 + 新 build 脚本替代，而不是只改一个接口文件。

3. 参考代码位置

• 接口定义：astra-sim-alibabacloud/astra-sim/system/AstraNetworkAPI.hh
• NS-3 实现：astra-sim-alibabacloud/astra-sim/network_frontend/ns3/
• 解析后端：astra-sim-alibabacloud/astra-sim/network_frontend/analytical/
• 物理网后端：astra-sim-alibabacloud/astra-sim/network_frontend/phynet/
• 可选快速路径包装：astra-sim-alibabacloud/astra-sim/system/fast-backend/

官方概念说明亦可对照：Connecting a Custom Network Backend (Astra-sim)（实现以本仓库 API 为准）。

EN

Reply to Issue #261: Replacing the network backend

1. Is this similar to Astra-sim’s “custom network backend”?

Yes, conceptually it is the same pattern. SimAI’s simulation stack is built on astra-sim-alibabacloud, where the upper layers (Sys, workload) talk to the network through AstraNetworkAPI. That matches the official Astra-sim idea: implement the network API and provide a backend-specific main / initialization wired to your discrete-event (or other) simulator.
Caveat: treat astra-sim/system/AstraNetworkAPI.hh in this repo as the source of truth—SimAI extends the API (e.g. sim_request / ncclFlowTag, BackendType) compared to the generic upstream documentation.

2. Which interfaces matter when swapping the backend?

Implement a subclass of AstraNetworkAPI and provide:

Category	API	Role
Lifecycle	sim_init, sim_finish, sim_time_resolution	Backend setup/teardown and time granularity
Time & scheduling	sim_get_time, sim_schedule	Advance simulated time; non-comm delays use sim_schedule callbacks
Communication	sim_send, sim_recv	Asynchronous: return after registering work; invoke msg_handler when the message completes
MPI-like	sim_comm_size, etc.	Match current stubs or implement real semantics as needed
SimAI extensions	get_backend_type, pass_front_end_report, get_BW_at_dimension	Override when queue routing or reporting differs

You also need a backend-specific entry point (see network_frontend/ns3/AstraSimNetwork.cc or network_frontend/analytical/AnalyticalAstra.cc): parse inputs, construct one network handler and one Sys per rank, call workload->fire(), then run the host simulator’s main event loop (e.g. NS-3’s Simulator::Run()).

Build integration: add a compile target analogous to the existing astra_ns3 / analytical builds and wire it in astra-sim-alibabacloud/build.sh / root scripts/build.sh. The NS-3 path copies ns-3-alibabacloud into extern/network_backend/ns3-interface; a new engine usually means a new extern layout + build script, not a one-line change.

3. Where to look in the tree

• API: astra-sim-alibabacloud/astra-sim/system/AstraNetworkAPI.hh
• NS-3 backend: astra-sim-alibabacloud/astra-sim/network_frontend/ns3/
• Analytical backend: astra-sim-alibabacloud/astra-sim/network_frontend/analytical/
• Physical-net backend: astra-sim-alibabacloud/astra-sim/network_frontend/phynet/
• Optional fast path wrapper: astra-sim-alibabacloud/astra-sim/system/fast-backend/

For conceptual background (implementation still follows this repo’s headers): Astra-sim: Connecting a Custom Network Backend.