+ 中文 ｜ English +
+ +# SimAI + +[](LICENSE) +[](https://ennanzhai.github.io/pub/nsdi25spring-simai.pdf) + # Latest News ### Recent Updates +- [2026/03] **SimAI 1.6 Released!** This release adds GPU memory modeling for inference simulation. Key features include: + + - **GPU Memory Module:** Accurate parameter counting and KV cache management for DeepSeek-V3-671B, Qwen3-MoE-235B, and Qwen3-Next-80B. See [SimAI 1.6 Tech Report](./docs/SimAI_1.6_Tech_Report.md). + - **PD-Separation Memory Planning:** Independent parameter memory and KV cache budget calculation for Prefill and Decode phases. See [memory_planner.py](./vidur-alibabacloud/vidur/scheduler/utils/memory_planner.py). + - **Improved Decode Time Estimation:** Linear interpolation replacing nearest-neighbor for AICB decode time prediction, with global cache for cross-run reuse. See [execution_time.py](./vidur-alibabacloud/vidur/entities/execution_time.py). + - **4-Scenario Test Suite:** End-to-end validation covering Qwen3-Next-80B, DeepSeek-671B, and Qwen3-MoE-235B. See [run_scenarios.sh](./vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh). + - **Bilingual Documentation:** Complete EN/ZH documentation system. See [English Docs](./docs/en/index.md) | [中文文档](./docs/zh/index.md). + - **GitHub Community Health Files:** Added [Issue Templates](./.github/ISSUE_TEMPLATE/), [PR Template](./.github/pull_request_template.md), [Code of Conduct](./CODE_OF_CONDUCT.md), [Security Policy](./SECURITY.md), [Contributing Guide](./CONTRIBUTING.md), and [Changelog](./CHANGELOG.md). + - **Code Quality:** Replaced print with logging, added bilingual docstrings, removed ~390 lines of dead code, standardized TODOs, and added type annotations across vidur-alibabacloud modules. + - [2025/12] **SimAI 1.5 Released!** This release brings end-to-end simulation for multi-request **inference** workloads. Key features include: - - - **Advanced Inference Simulation:** Model complex scenarios with Prefill/Decode separation. - - **Modern Model Support:** Now includes DeepSeek, Qwen3Moe and Qwen3Next. See [AICB's README](./aicb/README.md) for more detailed information. - - **Request Scheduling:** Request scheduling is now handled by a component adapted from Microsoft's [Vidur](https://github.com/microsoft/vidur). See [Vidur-Alibabacloud's README](./vidur-alibabacloud/README.md) for more detailed information. + + - **Advanced Inference Simulation:** Model complex scenarios with Prefill/Decode separation. + - **Modern Model Support:** Now includes DeepSeek, Qwen3Moe and Qwen3Next. See [AICB's README](./aicb/README.md) for more detailed information. + - **Request Scheduling:** Request scheduling is now handled by a component adapted from Microsoft's [Vidur](https://github.com/microsoft/vidur). See [Vidur-Alibabacloud's README](./vidur-alibabacloud/README.md) for more detailed information. - [2025/11] [AICB](https://github.com/aliyun/aicb/tree/master) now supports generating **prefill/decode** inference workloads for **DeepSeek**, **Qwen3-MoE** and **Qwen3-Next**. @@ -14,7 +33,8 @@ - [2025/06] The code of SimCCL is first released in the branch [SimCCL](https://github.com/aliyun/SimAI/tree/SimCCL) and will be released in SimCCL repository soon. -**We warmly welcome contributions from the community!** If you are interested in helping shape the future of SimAI, please feel free to open an issue to discuss your ideas or submit a pull request. +**We warmly welcome contributions from the community!** If you are interested in helping shape the future of SimAI, please feel free to open an issue to discuss your ideas or submit a pull request. +
🎯 Events & Community Engagement 🎯
@@ -29,6 +49,7 @@
| Date | Event | Location | Content | Type |
|:----------------:|:------------------------------------------------------------------------ |:----------------------- |:-------------------------------------------------------- |:-------------:|
+| Mar 16, 2026 | SimAI 1.6 | 🌐 Online | The release of SimAI 1.6 | 💻 Virtual |
| Dec 30, 2025 | SimAI 1.5 | 🌐 Online | The release of SimAI 1.5 | 💻 Virtual |
| Jun 4, 2025 | The first workshop of the SimAI community | 📍 Peking University | Three talks from community contributors | 🎓 On-site |
| May 24, 2025 | The 28th Chinasys workshop | 📍 Chongqing University | An invited talk about SimAI | 🎓 On-site |
@@ -44,6 +65,15 @@
---
+## Documentation
+
+| | |
+|---|---|
+| [English Documentation](./docs/en/index.md) | Full documentation in English |
+| [中文文档](./docs/zh/index.md) | 完整中文文档 |
+
+---
+
# Table of Contents
- [SimAI Overview](#simai-overview)
@@ -51,18 +81,17 @@
- [Components](#components)
- [Scenario](#scenario)
- [Citation](#citation)
-- [Usage](#usage)
+- [Quick Start](#quick-start)
- [Setup](#setup)
- - [From Source Code](#from-source-code)
- [Use SimAI-Analytical](#use-simai-analytical)
- [Use SimAI-Simulation](#use-simai-simulation)
- - [Use Vidur-AICB](#use-vidur-aicb)
+ - [Use Multi-requests Inference Simulation](#use-multi-requests-inference-simulation)
# SimAI Overview
## Introduction
-**SimAI** is the industry's first full-stack, high-precision **Sim**ulator for **AI** large-scale **\*\*inference\*\*** and **training**. It provides detailed modeling and simulation of the entire LLM training process, encompassing framework, collective communication, network layers, and more. This comprehensive approach offers end-to-end performance data, enabling researchers to:
+**SimAI** is the industry's first full-stack, high-precision **Sim**ulator for **AI** large-scale **inference** and **training**. It provides detailed modeling and simulation of the entire LLM training process, encompassing framework, collective communication, network layers, and more. This comprehensive approach offers end-to-end performance data, enabling researchers to:
- Analyze inference/training process details
- Evaluate the time consumption of AI tasks under specific conditions
@@ -86,7 +115,7 @@ SimAI --|--- SimCCL
|--- vidur-alibabacloud
-Building on pure simulation capabilities, SimAI has evolved into a versatile full-stack toolkit comprising four components ([aicb](https://github.com/aliyun/aicb), [SimCCL](https://github.com/aliyun/SimCCL), [astra-sim-alibabacloud](https://github.com/aliyun/SimAI/tree/master/astra-sim-alibabacloud), [ns-3-alibabacloud](https://github.com/aliyun/ns-3-alibabacloud)). These components can be combined in various ways to achieve different functionalities. Below, we present the six main usage scenarios for SimAI. We encourage users to explore even more possibilities with this powerful tool.
+Building on pure simulation capabilities, SimAI has evolved into a versatile full-stack toolkit comprising four components ([aicb](https://github.com/aliyun/aicb), [SimCCL](https://github.com/aliyun/SimCCL), [astra-sim-alibabacloud](https://github.com/aliyun/SimAI/tree/master/astra-sim-alibabacloud), [ns-3-alibabacloud](https://github.com/aliyun/ns-3-alibabacloud)). These components can be combined in various ways to achieve different functionalities. Below, we present the main usage scenarios for SimAI. We encourage users to explore even more possibilities with this powerful tool.
Below is the architecture diagram of the SimAI Simulator:
@@ -103,15 +132,15 @@ SimAI supports three major operation modes to meet different simulation requirem
**SimAI-Physical** *(Beta)* enables physical traffic generation for CPU RDMA cluster environments. This mode generates NCCL-like traffic patterns, allowing in-depth study of NIC behaviors during LLM training. It is currently in internal testing phase.
-| Scenario | Description | Component Combination |
-| -------------------------------------- | ------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| 1. AICB Test Suite | Run communication patterns on GPU clusters using AICB Test suite | [AICB](https://github.com/aliyun/aicb) |
-| 2. AICB/AIOB Workload | Model compute/communication patterns of **\*\*inference\*\*/training** process to generate workload | [AICB](https://github.com/aliyun/aicb) |
-| 3. Collective Comm Analyze | Break down collective communication operations into point-to-point communication sets | [SimCCL](https://github.com/aliyun/SimCCL) |
-| 4. Collective Comm w/o GPU | Perform RDMA collective communication traffic on non-GPU clusters | [AICB](https://github.com/aliyun/aicb) + [SimCCL](https://github.com/aliyun/SimCCL) + [astra-sim-alibabacloud](https://github.com/aliyun/SimAI/tree/master/astra-sim-alibabacloud)(physical) |
+| Scenario | Description | Component Combination |
+|----------------------------------------|-----------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| 1. AICB Test Suite | Run communication patterns on GPU clusters using AICB Test suite | [AICB](https://github.com/aliyun/aicb) |
+| 2. AICB/AIOB Workload | Model compute/communication patterns of **inference**/training process to generate workload | [AICB](https://github.com/aliyun/aicb) |
+| 3. Collective Comm Analyze | Break down collective communication operations into point-to-point communication sets | [SimCCL](https://github.com/aliyun/SimCCL) |
+| 4. Collective Comm w/o GPU | Perform RDMA collective communication traffic on non-GPU clusters | [AICB](https://github.com/aliyun/aicb) + [SimCCL](https://github.com/aliyun/SimCCL) + [astra-sim-alibabacloud](https://github.com/aliyun/SimAI/tree/master/astra-sim-alibabacloud)(physical) |
| 5. SimAI-Analytical | Conduct rapid AICB workload analysis and simulation on any server (ignoring underlying network details) | [AICB](https://github.com/aliyun/aicb) + [astra-sim-alibabacloud](https://github.com/aliyun/SimAI/tree/master/astra-sim-alibabacloud)(analytical) |
-| 6. SimAI-Simulation | Perform full simulation on any server | [AICB](https://github.com/aliyun/aicb) + [SimCCL](https://github.com/aliyun/SimCCL) + [astra-sim-alibabacloud](https://github.com/aliyun/SimAI/tree/master/astra-sim-alibabacloud)(simulation) + [ns-3-alibabacloud](https://github.com/aliyun/ns-3-alibabacloud) |
-| 7. Multi-requests Inference Simulation | Perform full multi-requests **inference** simulation using one GPU server | [AICB](https://github.com/aliyun/aicb) + [SimCCL](https://github.com/aliyun/SimCCL) + [vidur-alibabacloud](./vidur-alibabacloud) + [astra-sim-alibabacloud](https://github.com/aliyun/SimAI/tree/master/astra-sim-alibabacloud)(analytical/simulation) |
+| 6. SimAI-Simulation | Perform full simulation on any server | [AICB](https://github.com/aliyun/aicb) + [SimCCL](https://github.com/aliyun/SimCCL) + [astra-sim-alibabacloud](https://github.com/aliyun/SimAI/tree/master/astra-sim-alibabacloud)(simulation) + [ns-3-alibabacloud](https://github.com/aliyun/ns-3-alibabacloud) |
+| 7. Multi-requests Inference Simulation | Perform full multi-requests **inference** simulation using one GPU server | [AICB](https://github.com/aliyun/aicb) + [SimCCL](https://github.com/aliyun/SimCCL) + [vidur-alibabacloud](./vidur-alibabacloud) + [astra-sim-alibabacloud](https://github.com/aliyun/SimAI/tree/master/astra-sim-alibabacloud)(analytical/simulation) |
## Citation
@@ -125,15 +154,15 @@ We encourage innovative research and extensions based on SimAI. Welcome to join
# Quick Start
-Here are some simple examples, SimAI full tutorials can be found here: [**SimAI@Tutorial**](./docs/Tutorial.md), [**aicb@Tutorial**](https://github.com/aliyun/aicb/blob/master/training/tutorial.md), [SimCCL@Tutorial], [ns-3-alibabacloud@Tutorial]
+Here are some simple examples. SimAI full tutorials can be found here: [**SimAI@Tutorial**](./docs/Tutorial.md), [**aicb@Tutorial**](https://github.com/aliyun/aicb/blob/master/training/tutorial.md), [SimCCL@Tutorial], [ns-3-alibabacloud@Tutorial]
## Setup
-You can follow the instrucitons below to quickly set up the environtments and run SimAI
+You can follow the instructions below to quickly set up the environments and run SimAI.
### From Source Code
-The following code has been successfully tested on GCC/G++ 9.4.0, python 3.8.10 in Ubuntu 20.04
+The following code has been successfully tested on GCC/G++ 9.4.0, python 3.8.10 in Ubuntu 20.04.
You can use the official Ubuntu 20.04 image, and do not install ninja.
@@ -159,13 +188,13 @@ $ ./scripts/build.sh -c ns3
## Use SimAI-Analytical
```bash
-$ ./bin/SimAI_analytical -w example/workload_analytical.txt -g 9216 -g_p_s 8 -r test- -busbw example/busbw.yaml
+$ ./bin/SimAI_analytical -w example/workload_analytical.txt -g 9216 -g_p_s 8 -r test- -busbw example/busbw.yaml
```
-For calculating bus bandwidth autolly, please try the following command:
+For calculating bus bandwidth automatically, please try the following command:
```bash
-$ ./bin/SimAI_analytical -w ./example/workload_analytical.txt -g 9216 -nv 360 -nic 48.5 -n_p_s 8 -g_p_s 8 -r example-
+$ ./bin/SimAI_analytical -w ./example/workload_analytical.txt -g 9216 -nv 360 -nic 48.5 -n_p_s 8 -g_p_s 8 -r example-
```
## Use SimAI-Simulation
@@ -180,39 +209,57 @@ $ AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 16 -w ./example/microA
## Use Multi-requests Inference Simulation
-For detailed information, please refer to the [README](./vidur-alibabacloud/README.md) file in the `vidur-alibabacloud` directory. This module leverages AICB to profile the computation time of **inference** workloads. Due to its reliance on specific hardware-accelerated libraries like DeepGEMM and FlashMLA, it is exclusively compatible with NVIDIA GPUs based on the **Hopper (SM90)** and **Blackwell (SM100)** architectures.
+For detailed information, please refer to the [README](./vidur-alibabacloud/README.md) file in the `vidur-alibabacloud` directory. This module leverages AICB to profile the computation time of **inference** workloads. Due to its reliance on specific hardware-accelerated libraries like DeepGEMM and FlashMLA, it is exclusively compatible with NVIDIA GPUs based on the **Hopper (SM90)** and **Blackwell (SM100)** architectures.
-```shell
+```bash
# Build from Dockerfile
docker build -t image:latest .
-docker run --gpus all -it --rm image:latest
+docker run --gpus all -it --rm image:latest
```
-**Note**: please add `ENV FLASH_MLA_DISABLE_SM100=1` to Dockerfile if using Hopper GPUs.
+**Note:** Please add `ENV FLASH_MLA_DISABLE_SM100=1` to Dockerfile if using Hopper GPUs.
-# Acknowledgments
+To quickly validate all supported inference scenarios (Qwen3-Next-80B, DeepSeek-671B, Qwen3-MoE-235B), use the bundled 4-scenario test suite:
-A huge thanks to the following people and organizations who have contributed to this project:
+```bash
+# Prerequisites: conda activate vidur
+bash vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh --all
+# Or run a single scenario:
+bash vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh --scenario 1
+```
-- TianHao Fu (Peking University) and [TELOS-syslab](https://github.com/TELOS-syslab/),
+> **Prerequisites:** Requires `conda activate vidur` environment. See [Environment Setup](./vidur-alibabacloud/README.md#-environment-setup) for details.
+>
+> For detailed scenario configuration table and output file descriptions, see [Vidur-AlibabaCloud README](./vidur-alibabacloud/README.md#4-scenario-configuration).
-- Parth Parikh (KEYSIGHT),
+# Acknowledgments
-- Sarah-Michelle Hammer & Ziyi Wang (TU-Berlin),
+A huge thanks to the following people and organizations who have contributed to this project:
-- Xinyue Li (BUPT),
+- TianHao Fu (Peking University) and [TELOS-syslab](https://github.com/TELOS-syslab/)
+- Parth Parikh (KEYSIGHT)
+- Sarah-Michelle Hammer & Ziyi Wang (TU-Berlin)
+- Xinyue Li (BUPT)
+- Tong Chen (Zhejiang University)
+- Ming Wang (BUPT)
+- Tao Jiang (Institute of Computing Technology, Chinese Academy of Sciences)
-- Tong Chen (Zhejiang University),
+...and many other individual contributors from the community (See the [Contributors to aliyun/SimAI](https://github.com/aliyun/SimAI/graphs/contributors)).
-- Ming Wang (BUPT),
+We also thank Chenning Li (MIT CSAIL) who initiated the cooperation on integrating SimAI into [M4](https://github.com/netiken/m4), a new, innovative simulator.
-- Tao Jiang (Institute of Computing Technology, Chinese Academy of Sciences),
+**This project still welcomes more contributions and suggestions.**
-and many other individual contributors from the community (See the [Contributors to aliyun/SimAI · GitHub](https://github.com/aliyun/SimAI/graphs/contributors)).
+# Contributing
-We also thank Chenning Li (MIT CSAIL) who initiated the cooperation on integrating SimAI into [M4](https://github.com/netiken/m4), a new, innovative simulator.
+We welcome all contributions! Please read the following guides before getting started:
-**This project still welcomes more contributions and suggestions**.
+| | |
+|---|---|
+| [Contributing Guide](./CONTRIBUTING.md) | How to submit issues and pull requests |
+| [Security Policy](./SECURITY.md) | How to report security vulnerabilities |
+| [Code of Conduct](./CODE_OF_CONDUCT.md) | Our community standards |
+| [Changelog](./CHANGELOG.md) | Version history from v1.5 onwards |
# Contact us
@@ -224,5 +271,3 @@ Welcome to join the SimAI community chat groups, with the DingTalk group on the
-
-
diff --git a/README_CN.md b/README_CN.md new file mode 100644 index 00000000..1302de50 --- /dev/null +++ b/README_CN.md @@ -0,0 +1,273 @@ +
+ 中文 ｜ English +
+ +# SimAI + +[](LICENSE) +[](https://ennanzhai.github.io/pub/nsdi25spring-simai.pdf) + +# 最新动态 + +### 近期更新 + +- [2026/03] **SimAI 1.6 正式发布！** 本版本新增推理仿真的 GPU 内存建模能力。主要特性包括： + + - **GPU 内存计算模块：** 支持 DeepSeek-V3-671B、Qwen3-MoE-235B、Qwen3-Next-80B 的精确参数计数与 KV Cache 管理。详见 [SimAI 1.6 技术报告](./docs/SimAI_1.6_Tech_Report_CN.md)。 + - **PD 分离内存规划：** Prefill 与 Decode 阶段独立的参数内存和 KV Cache 预算计算。详见 [memory_planner.py](./vidur-alibabacloud/vidur/scheduler/utils/memory_planner.py)。 + - **Decode 时间估算改进：** 首尾线性插值替代最近邻的 AICB decode 时间预测，全局缓存支持跨运行复用。详见 [execution_time.py](./vidur-alibabacloud/vidur/entities/execution_time.py)。 + - **4 场景端到端测试：** 覆盖 Qwen3-Next-80B、DeepSeek-671B、Qwen3-MoE-235B 的完整验证套件。详见 [run_scenarios.sh](./vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh)。 + - **双语文档体系：** 全套 EN/ZH 文档系统。详见 [English Docs](./docs/en/index.md) | [中文文档](./docs/zh/index.md)。 + - **GitHub 社区规范文件：** 新增 [Issue 模板](./.github/ISSUE_TEMPLATE/)、[PR 模板](./.github/pull_request_template.md)、[行为准则](./CODE_OF_CONDUCT.md) ([中文](./CODE_OF_CONDUCT_CN.md))、[安全政策](./SECURITY.md) ([中文](./SECURITY_CN.md))、[贡献指南](./CONTRIBUTING.md) ([中文](./CONTRIBUTING.zh-CN.md)) 和 [更新日志](./CHANGELOG.md) ([中文](./CHANGELOG_CN.md))。 + - **代码质量提升：** logging 替换 print 输出、双语 docstring、清理 ~390 行死代码、TODO 规范化、类型标注补全。 + +- [2025/12] **SimAI 1.5 正式发布！** 本版本新增对多请求**推理**工作负载的端到端仿真支持，主要特性包括： + + - **高级推理仿真：** 支持 Prefill/Decode 分离等复杂场景建模。 + - **主流模型支持：** 新增 DeepSeek、Qwen3Moe 和 Qwen3Next 模型。详见 [AICB README](./aicb/README.md)。 + - **请求调度：** 请求调度组件基于微软 [Vidur](https://github.com/microsoft/vidur) 适配，详见 [Vidur-Alibabacloud README](./vidur-alibabacloud/README_CN.md)。 + +- [2025/11] [AICB](https://github.com/aliyun/aicb/tree/master) 新增对 **DeepSeek**、**Qwen3-MoE** 和 **Qwen3-Next** 的 **prefill/decode** 推理工作负载生成支持。 + +- [2025/09] [AICB](https://github.com/aliyun/aicb/tree/master) 新增 DeepSeek 训练工作负载生成支持。感谢 [@parthpower](https://github.com/parthpower) 的贡献。 + +- [2025/06] SimCCL 代码首次在 [SimCCL](https://github.com/aliyun/SimAI/tree/SimCCL) 分支发布，后续将在独立仓库正式开源。 + +**欢迎社区贡献！** 如有想法，欢迎提交 Issue 讨论或发起 Pull Request。 + +
+🎯 活动与社区 🎯
+
+### 📅 即将举办
+
+| 日期 | 活动 | 地点 | 内容 | 形式 |
+|:----:|:----- |:-------- |:------- |:----:|
+| -- | | | | |
+
+### 🌟 往期活动
+
+| 日期 | 活动 | 地点 | 内容 | 形式 |
+|:----------------:|:------------------------------------------------------------------------ |:----------------------- |:-------------------------------------------------------- |:-------------:|
+| Mar 16, 2026 | SimAI 1.6 | 🌐 线上 | SimAI 1.6 正式发布 | 💻 线上直播 |
+| Dec 30, 2025 | SimAI 1.5 | 🌐 线上 | SimAI 1.5 正式发布 | 💻 线上直播 |
+| Jun 4, 2025 | SimAI 社区第一届研讨会 | 📍 北京大学 | 三场社区贡献者演讲 | 🎓 线下 |
+| May 24, 2025 | 第 28 届 Chinasys 研讨会 | 📍 重庆大学 | SimAI 受邀演讲 | 🎓 线下 |
+| Dec 27, 2024 | SimAI 技术分享 | 📍 北京航空航天大学 | SimAI 技术分享与交流 | 🎓 线下 |
+| Dec 6, 2024 | 香港科技大学技术研讨会 | 📍 香港科技大学（广州） | SimAI 技术分享与交流 | 🎓 线下 |
+| Dec 5, 2024 | [Bench'24 会议](https://mp.weixin.qq.com/s/STic_E12xMhZRxhzK9wRnw) | 📍 广州 | SimAI 教程与深度技术专场 | 🎓 线下 |
+| Nov 26, 2024 | SimAI 社区直播 | 🌐 线上 | 互动技术交流与演示（400+ 参与者） | 💻 线上直播 |
+| Nov 15, 2024 | 技术研讨会 | 📍 千岛湖 | SimAI 线下技术交流 | 🎯 线下 |
+| Oct 18, 2024 | 嘉宾讲座 | 📍 复旦大学 | SimAI 教程与公开课 | 🎓 线下 |
+| Sept 24-26, 2024 | CCF HPC China 2024 | 📍 武汉 | SimAI 介绍与技术报告 | 🎤 会议 |
+
+
+
+---
+
+## 文档
+
+| | |
+|---|---|
+| [English Documentation](./docs/en/index.md) | Full documentation in English |
+| [中文文档](./docs/zh/index.md) | 完整中文文档 |
+
+---
+
+# 目录
+
+- [SimAI 概述](#simai-概述)
+ - [简介](#简介)
+ - [组件](#组件)
+ - [应用场景](#应用场景)
+ - [引用](#引用)
+- [快速开始](#快速开始)
+ - [环境搭建](#环境搭建)
+ - [使用 SimAI-Analytical](#使用-simai-analytical)
+ - [使用 SimAI-Simulation](#使用-simai-simulation)
+ - [使用多请求推理仿真](#使用多请求推理仿真)
+
+# SimAI 概述
+
+## 简介
+
+**SimAI** 是业界首个全栈高精度 AI 大规模**推理**与**训练**模拟器（**Sim**ulator for **AI**）。它对 LLM 训练全流程进行详细建模和仿真，涵盖框架、集合通信、网络层等，提供端到端的性能数据，帮助研究人员：
+
+- 分析推理/训练过程细节
+- 评估特定条件下 AI 任务的耗时
+- 评估各类算法优化带来的 E2E 性能收益，包括：
+ - 框架参数配置
+ - 集合通信算法
+ - NCCL 环境变量
+ - 网络传输协议
+ - 拥塞控制算法
+ - 自适应路由算法
+ - 扩展/集合网络拓扑调整
+ - ……
+
+## 组件
+
++ |--- AICB +SimAI --|--- SimCCL + |--- astra-sim-alibabacloud + |--- ns-3-alibabacloud + |--- vidur-alibabacloud ++ +在纯仿真能力基础上，SimAI 已演进为一个由四个组件（[aicb](https://github.com/aliyun/aicb)、[SimCCL](https://github.com/aliyun/SimCCL)、[astra-sim-alibabacloud](https://github.com/aliyun/SimAI/tree/master/astra-sim-alibabacloud)、[ns-3-alibabacloud](https://github.com/aliyun/ns-3-alibabacloud)）构成的全栈工具套件。这些组件可以灵活组合以实现不同功能。我们鼓励用户探索更多可能性。 + +下图为 SimAI 模拟器架构图： + + +astra-sim-alibabacloud 基于 [astra-sim](https://github.com/astra-sim/astra-sim/tree/ASTRA-sim-1.0) 扩展开发。感谢 astra-sim 团队的优秀工作和开源贡献。我们在其基础上集成了 NCCL 算法并添加了若干新特性。 + +## 应用场景 + +SimAI 支持三种主要运行模式： + +**SimAI-Analytical** 通过使用总线带宽（busbw）抽象网络通信细节来估算集合通信时间，实现快速仿真。目前支持用户自定义 busbw，自动计算 busbw 功能即将推出。 + +**SimAI-Simulation** 提供基于细粒度网络通信建模的全栈仿真。利用 NS-3 或其他网络模拟器（当前 NS-3 已开源）实现对所有通信行为的详细仿真，力求高保真还原真实训练环境。 + +**SimAI-Physical** *(Beta)* 支持在 CPU RDMA 集群环境下生成物理流量，通过生成类 NCCL 的流量模式深入研究 LLM 训练中的 NIC 行为。当前处于内测阶段。 + +| 场景 | 描述 | 组件组合 | +|------|------|----------| +| 1. AICB 测试套件 | 在 GPU 集群上使用 AICB 测试套件运行通信模式 | [AICB](https://github.com/aliyun/aicb) | +| 2. AICB/AIOB 工作负载 | 建模**推理**/训练过程的计算/通信模式以生成工作负载 | [AICB](https://github.com/aliyun/aicb) | +| 3. 集合通信分析 | 将集合通信操作分解为点对点通信集合 | [SimCCL](https://github.com/aliyun/SimCCL) | +| 4. 无 GPU 集合通信 | 在非 GPU 集群上执行 RDMA 集合通信流量 | [AICB](https://github.com/aliyun/aicb) + [SimCCL](https://github.com/aliyun/SimCCL) + [astra-sim-alibabacloud](https://github.com/aliyun/SimAI/tree/master/astra-sim-alibabacloud)(physical) | +| 5. SimAI-Analytical | 在任意服务器上快速进行 AICB 工作负载分析与仿真（忽略底层网络细节） | [AICB](https://github.com/aliyun/aicb) + [astra-sim-alibabacloud](https://github.com/aliyun/SimAI/tree/master/astra-sim-alibabacloud)(analytical) | +| 6. SimAI-Simulation | 在任意服务器上进行全栈仿真 | [AICB](https://github.com/aliyun/aicb) + [SimCCL](https://github.com/aliyun/SimCCL) + [astra-sim-alibabacloud](https://github.com/aliyun/SimAI/tree/master/astra-sim-alibabacloud)(simulation) + [ns-3-alibabacloud](https://github.com/aliyun/ns-3-alibabacloud) | +| 7. 多请求推理仿真 | 在单 GPU 服务器上进行多请求**推理**全栈仿真 | [AICB](https://github.com/aliyun/aicb) + [SimCCL](https://github.com/aliyun/SimCCL) + [vidur-alibabacloud](./vidur-alibabacloud) + [astra-sim-alibabacloud](https://github.com/aliyun/SimAI/tree/master/astra-sim-alibabacloud)(analytical/simulation) | + +## 引用 + +SimAI 论文已被 NSDI'25 Spring 接收，详情请参阅： + +*SimAI: Unifying Architecture Design and Performance Tuning for Large-Scale Large Language Model Training with Scalability and Precision.* + +[[pdf](https://ennanzhai.github.io/pub/nsdi25spring-simai.pdf)] / [[slides](./docs/SimAI_Intro_Online.pdf)] / [[video](https://n.dingtalk.com/dingding/live-room/index.html?roomId=OF5BkBUXVxmgsK7x&liveUuid=305736cd-aa70-498b-8003-2b471a53decd)] + +欢迎基于 SimAI 开展创新研究和功能扩展。欢迎加入社区群或通过邮件联系我们交流，我们可提供技术支持。 + +# 快速开始 + +以下为简单示例。完整教程请参见：[**SimAI@Tutorial**](./docs/Tutorial.md)、[**aicb@Tutorial**](https://github.com/aliyun/aicb/blob/master/training/tutorial.md)、[SimCCL@Tutorial]、[ns-3-alibabacloud@Tutorial] + +## 环境搭建 + +请按照以下步骤快速搭建环境并运行 SimAI。 + +### 从源码安装 + +以下步骤已在 Ubuntu 20.04 的 GCC/G++ 9.4.0、python 3.8.10 环境下验证。 + +可使用官方 Ubuntu 20.04 镜像，**不要安装 ninja**。 + +（对于工作负载生成场景，推荐直接使用 NGC 容器镜像。） + +```bash +# 克隆仓库 +$ git clone https://github.com/aliyun/SimAI.git +$ cd ./SimAI/ + +# 初始化子模块 +$ git submodule update --init --recursive +# 更新到最新提交 +$ git submodule update --remote + +# 编译 SimAI-Analytical +$ ./scripts/build.sh -c analytical + +# 编译 SimAI-Simulation (ns3) +$ ./scripts/build.sh -c ns3 +``` + +## 使用 SimAI-Analytical + +```bash +$ ./bin/SimAI_analytical -w example/workload_analytical.txt -g 9216 -g_p_s 8 -r test- -busbw example/busbw.yaml +``` + +若需自动计算总线带宽，请尝试： + +```bash +$ ./bin/SimAI_analytical -w ./example/workload_analytical.txt -g 9216 -nv 360 -nic 48.5 -n_p_s 8 -g_p_s 8 -r example- +``` + +## 使用 SimAI-Simulation + +```bash +# 生成网络拓扑 +$ python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py -topo Spectrum-X -g 128 -gt A100 -bw 100Gbps -nvbw 2400Gbps + +# 运行仿真 +$ AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 16 -w ./example/microAllReduce.txt -n ./Spectrum-X_128g_8gps_100Gbps_A100 -c astra-sim-alibabacloud/inputs/config/SimAI.conf +``` + +## 使用多请求推理仿真 + +详情请参见 `vidur-alibabacloud` 目录下的 [README](./vidur-alibabacloud/README_CN.md)。该模块利用 AICB 对**推理**工作负载的计算时间进行 profiling。由于依赖 DeepGEMM 和 FlashMLA 等特定硬件加速库，目前仅兼容基于 **Hopper（SM90）** 和 **Blackwell（SM100）** 架构的 NVIDIA GPU。 + +```bash +# 从 Dockerfile 构建 +docker build -t image:latest . +docker run --gpus all -it --rm image:latest +``` + +**注意：** 若使用 Hopper GPU，请在 Dockerfile 中添加 `ENV FLASH_MLA_DISABLE_SM100=1`。 + +如需快速验证所有支持的推理场景（Qwen3-Next-80B、DeepSeek-671B、Qwen3-MoE-235B），可使用内置的四场景测试套件： + +```bash +# 前置条件：conda activate vidur +bash vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh --all +# 或单独运行某个场景： +bash vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh --scenario 1 +``` + +> **前置条件：** 需先激活 `conda activate vidur` 环境。详见 [环境配置](./vidur-alibabacloud/README_CN.md#-环境配置)。 +> +> 完整场景配置表与输出文件说明请参见 [Vidur-AlibabaCloud README](./vidur-alibabacloud/README_CN.md#四场景配置说明)。 + +# 致谢 + +衷心感谢以下人员和机构对本项目的贡献： + + +- TianHao Fu (Peking University) and [TELOS-syslab](https://github.com/TELOS-syslab/) +- Parth Parikh (KEYSIGHT) +- Sarah-Michelle Hammer & Ziyi Wang (TU-Berlin) +- Xinyue Li (BUPT) +- Tong Chen (Zhejiang University) +- Ming Wang (BUPT) +- Tao Jiang (Institute of Computing Technology, Chinese Academy of Sciences) + +……以及众多来自社区的个人贡献者（详见 [Contributors to aliyun/SimAI](https://github.com/aliyun/SimAI/graphs/contributors)）。 + +同时感谢 Chenning Li（MIT CSAIL）发起了将 SimAI 集成到 [M4](https://github.com/netiken/m4) 的合作——M4 是一个新型创新模拟器。 + +**本项目持续欢迎更多贡献与建议。** + +# 贡献指南 + +欢迎参与贡献！开始前请阅读以下指引： + +| | | +|---|---| +| [贡献指南](./CONTRIBUTING.zh-CN.md) | 如何提交 Issue 和 Pull Request | +| [安全政策](./SECURITY_CN.md) | 如何报告安全漏洞 | +| [行为准则](./CODE_OF_CONDUCT_CN.md) | 社区行为规范 | +| [更新日志](./CHANGELOG_CN.md) | v1.5 起的版本历史 | + +# 联系我们 + +如有任何问题，欢迎发送邮件至：Gang Lu（yunding.lg@alibaba-inc.com）、Feiyang Xue（xuefeiyang.xfy@alibaba-inc.com）或 Qingxu Li（qingxu.lqx@alibaba-inc.com）。 + +欢迎加入 SimAI 社区交流群，左侧为钉钉群，右侧为微信群。 + +
+
+
+
diff --git a/SECURITY.md b/SECURITY.md
new file mode 100644
index 00000000..aa9e066e
--- /dev/null
+++ b/SECURITY.md
@@ -0,0 +1,47 @@
+
+
++ 中文 ｜ English +
+ +# Security Policy + +## Reporting a Vulnerability + +The SimAI team takes security issues seriously. We appreciate your efforts to responsibly disclose any security vulnerabilities you find. + +**Please do NOT report security vulnerabilities through public GitHub issues.** + +Instead, please report them via email to: + +- Gang Lu: yunding.lg@alibaba-inc.com +- Feiyang Xue: xuefeiyang.xfy@alibaba-inc.com +- Qingxu Li: qingxu.lqx@alibaba-inc.com + +Please include the following information in your report: + +- Description of the vulnerability +- Steps to reproduce the issue +- Potential impact +- Suggested fix (if any) + +## Response Timeline + +- We will acknowledge receipt of your vulnerability report within **3 business days**. +- We will provide a more detailed response within **10 business days**, indicating the next steps for handling your report. +- We will keep you informed of the progress towards a fix and full announcement. + +## Supported Versions + +| Version | Supported | +|---------|--------------------| +| 1.5.x | :white_check_mark: | +| < 1.5 | :x: | + +## Disclosure Policy + +When we receive a security bug report, we will: + +1. Confirm the problem and determine the affected versions. +2. Audit code to find any similar problems. +3. Prepare fixes and release them as soon as possible. + +Thank you for helping keep SimAI and its users safe! diff --git a/SECURITY_CN.md b/SECURITY_CN.md new file mode 100644 index 00000000..87c911be --- /dev/null +++ b/SECURITY_CN.md @@ -0,0 +1,47 @@ ++ 中文 ｜ English +
+ +# 安全政策 + +## 报告漏洞 + +SimAI 团队高度重视安全问题。我们感谢您以负责任的方式披露所发现的安全漏洞。 + +**请勿通过公开的 GitHub Issue 报告安全漏洞。** + +请通过电子邮件将漏洞报告发送至以下联系人： + +- Gang Lu：yunding.lg@alibaba-inc.com +- Feiyang Xue：xuefeiyang.xfy@alibaba-inc.com +- Qingxu Li：qingxu.lqx@alibaba-inc.com + +报告中请包含以下信息： + +- 漏洞描述 +- 复现步骤 +- 潜在影响 +- 修复建议（如有） + +## 响应时间线 + +- 我们将在 **3 个工作日**内确认收到您的漏洞报告。 +- 我们将在 **10 个工作日**内提供更详细的回复，说明后续处理步骤。 +- 我们将持续向您通报修复进展及公告发布情况。 + +## 支持的版本 + +| 版本 | 是否支持 | +|---------|-----------------------| +| 1.5.x | :white_check_mark: | +| < 1.5 | :x: | + +## 披露政策 + +收到安全漏洞报告后，我们将： + +1. 确认问题并确定受影响的版本。 +2. 对代码进行审查，排查类似问题。 +3. 尽快准备并发布修复补丁。 + +感谢您帮助保障 SimAI 及其用户的安全！ diff --git a/docs/SimAI_1.6_Tech_Report.md b/docs/SimAI_1.6_Tech_Report.md new file mode 100644 index 00000000..b00809b7 --- /dev/null +++ b/docs/SimAI_1.6_Tech_Report.md @@ -0,0 +1,429 @@ ++ 中文 ｜ English +
+ +# SimAI 1.6 Technical Report + +> This report covers all features from SimAI 1.5 as well as the new enhancements introduced in SimAI 1.6. + +## 1. Overview + +**SimAI** is the industry's first full-stack, high-precision **Sim**ulator for **AI** large-scale inference and training, open-sourced by Alibaba Cloud. SimAI provides detailed modeling and simulation of the entire LLM inference and training process, encompassing the framework layer, collective communication layer, and network transport layer, delivering end-to-end performance data. The SimAI paper was accepted by NSDI'25 Spring [1]. + +SimAI 1.6 builds upon SimAI 1.5 with further enhancements, primarily introducing the **GPU Memory Calculation Module** (supporting accurate parameter counting and KV cache management for DeepSeek-V3-671B, Qwen3-MoE-235B, and Qwen3-Next-80B), a **4-Scenario End-to-End Test Suite**, and comprehensive code quality improvements (bilingual documentation, logging system, dead code cleanup, etc.). + +### Component Overview + +``` + |--- AICB (Workload generation & compute profiling) +SimAI --|--- SimCCL (Collective communication algorithm analysis) + |--- astra-sim-alibabacloud (Simulation engine: Analytical / Simulation / Physical) + |--- ns-3-alibabacloud (NS-3 network backend) + |--- vidur-alibabacloud (Multi-request inference scheduling & memory management) +``` + +--- + +## 2. Key Milestones + +The following are the key development events from November 2025 to March 2026: + +| Date | Event | Description | +|------|-------|-------------| +| 2025/11 | AICB PR [#58](https://github.com/aliyun/aicb/pull/58) | AICB adds inference workload generation with prefill/decode phase separation, supporting DeepSeek, Qwen3-MoE, and Qwen3-Next | +| 2025/12 | AICB PR [#60](https://github.com/aliyun/aicb/pull/60) | AICB further update, refining inference workload generation | +| 2025/12 | SimAI PR [#203](https://github.com/aliyun/SimAI/pull/203) | SimAI 1.5 core update: end-to-end inference simulation, PD disaggregation, Vidur scheduling integration, modern model support | +| 2025/12 | ns-3 commit [7e3cb5b](https://github.com/aliyun/ns-3-alibabacloud/commit/7e3cb5b88c99abcb582c5abc3919484a4805111b) | ns-3-alibabacloud README documentation enhancement with detailed NS3 backend modifications | +| 2026/01 | Memory module commits | Completed accurate memory calculation for DeepSeek-V3-671B, Qwen3-Next-80B, and Qwen3-MoE-235B | +| 2026/02 | PD disaggregation memory planning | Implemented independent parameter memory and KV cache budget calculation for Prefill/Decode phases | +| 2026/03 | Code quality improvements | Comprehensive bilingual comments/docs/logs, dead code cleanup, TODO standardization, type annotations | + +--- + +## 3. End-to-End Inference Simulation + +SimAI supports complete multi-request LLM inference simulation with the following core features: + +### 3.1 Prefill-Decode (PD) Disaggregation Architecture + +The inference process is divided into two phases: + +- **Prefill phase**: Processes all input prompt tokens and generates the first output token (compute-intensive) +- **Decode phase**: Autoregressively generates subsequent output tokens one at a time (memory-bandwidth-intensive) + +PD disaggregation allows deploying Prefill and Decode phases on different GPU nodes, enabling: +- Elastic resource allocation (Prefill nodes can be configured with more compute, Decode nodes with more memory) +- Performance isolation (avoiding resource contention between Prefill and Decode) +- Flexible P:D node ratio configuration (via `--replica_config_pd_node_ratio`) + +This design was inspired by [splitwise-sim](https://github.com/Mutinifni/splitwise-sim) [6]. + +### 3.2 Multi-Request Inference Scheduling + +The request scheduling component is adapted from Microsoft's [Vidur](https://github.com/microsoft/vidur) [5] (vidur-alibabacloud), supporting the following scheduling strategies: + +| Scheduler Type | Level | Description | +|---------------|-------|-------------| +| `split_wise` | Global | Global scheduling for PD disaggregation, dispatching requests to Prefill and Decode replicas | +| `lor` | Global | Least Outstanding Requests, dispatching to the least-loaded replica | +| `round_robin` | Global | Round-robin dispatch | +| `sarathi` | Per-replica | Intra-replica batch scheduling | +| `split_wise` | Per-replica | Per-replica scheduling for PD disaggregation | + +### 3.3 Flexible Parallelism + +Supports combinations of multiple parallelism strategies: + +- **Data Parallel (DP)** — via `--cluster_config_num_replicas` +- **Tensor Parallel (TP)** — via `--replica_config_tensor_parallel_size` +- **Pipeline Parallel (PP)** — via `--replica_config_num_pipeline_stages` +- **Expert Parallel (EP)** — via `--replica_config_expert_model_parallel_size` + +Works for both dense and MoE (Mixture-of-Experts) models. + +### 3.4 Multiple Execution-Time Prediction Backends + +| Backend | Description | +|---------|-------------| +| **AICB/AIOB** | Partially supports compute kernels and TP/DP/PP/EP communication size modeling for DeepSeek-V3-671B, Qwen3-MoE-235B, Qwen3-Next-80B | +| **SimAI Simulation** | SimAI NS-3-based full-stack network simulation (currently supports TP) | +| **SimAI Analytical** | SimAI analytical performance model (currently supports TP) | +| **Native Vidur** | Original Vidur backend, supports TP, DP, PP | + +--- + +## 4. Modern Model Support + +SimAI 1.6 supports the following three state-of-the-art MoE large models, with configuration files located in `vidur-alibabacloud/data/hf_configs/`: + +### 4.1 DeepSeek-V3-671B + +| Attribute | Value | +|-----------|-------| +| Total Layers | 61 | +| Attention Type | MLA (Multi-head Latent Attention) | +| Attention Heads | 128 | +| Hidden Size | 7168 | +| KV LoRA Rank | 512 | +| Q LoRA Rank | 1536 | +| QK RoPE Head Dim | 64 | +| QK NoPE Head Dim | 128 | +| V Head Dim | 128 | +| MoE Routed Experts | 256 | +| Experts Per Token | 8 | +| Shared Experts | 1 | +| Dense Layers (first 3) | Fixed activation of 8 routed experts + 1 shared expert | +| Sparse Layers (layers 3-60) | Dynamically select 8 from 256 routed experts + 1 shared expert | + +Configuration file: `data/hf_configs/deepseek_v3_config.json` + +### 4.2 Qwen3-MoE-235B + +| Attribute | Value | +|-----------|-------| +| Total Layers | 94 | +| Attention Type | MHA/GQA | +| Attention Heads | 64 | +| KV Heads | 4 | +| Hidden Size | 4096 | +| Head Dim | 128 | +| MoE Routed Experts | 128 | +| Experts Per Token | 8 | +| MoE Intermediate Size | 1536 | + +Configuration file: `data/hf_configs/qwen3_moe_config.json` + +### 4.3 Qwen3-Next-80B + +| Attribute | Value | +|-----------|-------| +| Total Layers | 48 | +| Attention Type | Hybrid (full + linear attention, alternating every 4 layers) | +| Full Attention Heads | 16 | +| KV Heads | 2 | +| Hidden Size | 2048 | +| Head Dim | 256 | +| Linear Attention Key Heads | 16 | +| Linear Attention Value Heads | 32 | +| MoE Routed Experts | 512 | +| Experts Per Token | 10 | +| MoE Intermediate Size | 512 | + +Configuration file: `data/hf_configs/qwen3-next-80B-A3B_config.json` + +--- + +## 5. GPU Memory Calculation Module + +This is the core new feature in SimAI 1.6. The module provides accurate GPU memory estimation for inference simulation, covering model parameter memory, KV cache memory, and maximum batch size calculation, with separate memory budget computation for Prefill and Decode phases under PD disaggregation. + +### 5.1 Parameter Counting (ParamCounter) + +**File path**: `vidur-alibabacloud/vidur/utils/param_counter.py` + +ParamCounter supports per-layer and per-device parameter counting, returning a triple `(total_params, prefill_params, decode_params)` under PD disaggregation. + +#### MLA Parameters (DeepSeek-V3-671B) + +Per-layer MLA parameter components: + +- **Q LoRA down-projection**: `wq_down = hidden_size * q_lora_rank` = 7168 * 1536 +- **Q LoRA up-projection**: `wq_up = q_lora_rank * num_attention_heads * qk_head_dim` = 1536 * 128 * 192, where `qk_head_dim = qk_nope_head_dim + qk_rope_head_dim = 128 + 64 = 192` +- **KV LoRA down-projection**: `wkv_down = hidden_size * kv_lora_rank` = 7168 * 512 +- **KV LoRA up-projection**: `wkv_up = kv_lora_rank * num_attention_heads * (qk_nope_head_dim + v_head_dim)` = 512 * 128 * 256 +- **Output projection**: `wo = hidden_size * num_attention_heads * v_head_dim` = 7168 * 128 * 128 + +Under FP8 quantization, each parameter element uses 1 byte; under FP16/BF16, each uses 2 bytes. + +References: [3] [4] + +#### MHA/GQA Parameters (Qwen3-MoE-235B) + +Per-layer MHA parameters: + +``` +wq = hidden_size * num_attention_heads * head_dim +wk = hidden_size * num_key_value_heads * head_dim +wv = hidden_size * num_key_value_heads * head_dim +wo = hidden_size * num_attention_heads * head_dim +total = (wq + wk + wv + wo) * bytes_per_element +``` + +#### Linear Attention Parameters (Qwen3-Next-80B) + +Qwen3-Next-80B uses a hybrid attention architecture, alternating between full attention and linear (GDN) attention every 4 layers. Linear attention layers use independent key/value head configurations (`linear_key_head_dim`, `linear_num_key_heads`, etc.). + +#### MoE Expert Parameters + +Per-expert FFN parameters (3 weight matrices W1, W2, W3): + +``` +expert_params = 3 * hidden_size * moe_intermediate_size * bytes_per_element +``` + +#### PD Disaggregation Parameter Calculation + +Under PD disaggregation, the expert parallelism (EP) may differ between Prefill and Decode clusters: + +- **Prefill cluster**: Uses `prefill_world_size` as EP, experts per device = `num_routed_experts / prefill_world_size` +- **Decode cluster**: Uses `decode_world_size` as EP, experts per device = `num_routed_experts / decode_world_size` + +This results in different parameter memory for Prefill and Decode clusters, which in turn affects their respective available KV cache capacity. + +### 5.2 KV Cache Memory Management + +**File path**: `vidur-alibabacloud/vidur/scheduler/utils/memory_planner.py`, `vidur-alibabacloud/vidur/entities/replica.py` + +#### MHA/GQA KV Cache Calculation + +``` +kv_cache_per_token = 2 * num_kv_heads * head_dim * num_layers * bytes_per_element +``` + +The factor of 2 represents the K (Key) and V (Value) caches. + +#### MLA KV Cache Calculation (DeepSeek-V3-671B) + +The MLA architecture uses compressed KV representations. Unlike MHA which stores separate K and V caches, MLA stores a single compressed latent vector (`kv_lora_rank`) that jointly encodes K and V, plus the RoPE position keys (`qk_rope_head_dim`). Per-token KV cache size: + +``` +kv_cache_per_token = (kv_lora_rank + qk_rope_head_dim) * num_layers * bytes_per_element +``` + +Where `kv_lora_rank = 512` and `qk_rope_head_dim = 64`. Compared to MHA's per-token cache of `2 * num_kv_heads * head_dim` = 2 * 128 * 128 = 32768 elements, MLA reduces this to 576 elements — a **~57x** reduction. + +#### Per-Request KV Cache Tracking + +The `Replica` entity (`vidur/entities/replica.py`) maintains the following state: + +- `_allocated_kv_cache_memory`: Currently allocated KV cache memory (bytes) +- `_max_kv_cache_memory`: Maximum KV cache capacity (computed on first call by MemoryPlanner) +- `_kv_cache_allocation_map`: Per-request KV cache allocation mapping + +Supported operations: +- `allocate_request_kv_cache_memory(request, num_blocks, block_size)` — Allocate KV cache for a request +- `release_request_kv_cache_memory(request)` — Release KV cache for a completed request +- `get_remaining_kv_cache_capacity()` — Query remaining KV cache capacity and serviceable request count + +### 5.3 MemoryPlanner + +**File path**: `vidur-alibabacloud/vidur/scheduler/utils/memory_planner.py` + +MemoryPlanner is the central component for memory management, with the following calculation flow: + +1. **Compute available GPU memory**: `available_memory = total_GPU_memory * (1 - memory_margin_fraction)` +2. **Get model parameter memory**: Computed via ParamCounter; under PD disaggregation returns `(total, prefill, decode)` triple +3. **Compute KV cache available memory**: `kv_cache_available = available_memory - param_memory` +4. **Compute maximum concurrent requests**: `max_requests = kv_cache_available / kv_cache_per_request` + +Under PD disaggregation: +- Prefill replicas use `prefill_param_mem` for KV cache budget calculation +- Decode replicas use `decode_param_mem` for KV cache budget calculation + +Includes OOM detection: when parameter memory exceeds available memory, error messages are output with suggestions to increase TP/EP, use larger GPUs, or enable FP8 quantization. + +--- + +## 6. AICB Inference Workload Generation + +[AICB](https://github.com/aliyun/aicb) introduces inference workload generation capabilities (PR [#58](https://github.com/aliyun/aicb/pull/58), [#60](https://github.com/aliyun/aicb/pull/60)), with key features: + +- **Prefill/Decode phase separation**: Generates separate compute and communication workloads for Prefill and Decode phases +- **Compute kernel profiling**: Relies on the following hardware-accelerated libraries (requires Hopper SM90 or Blackwell SM100 GPUs): + - [DeepGEMM](https://github.com/deepseek-ai/DeepGEMM) — FP8 matrix multiplication + - [FlashMLA](https://github.com/deepseek-ai/FlashMLA) — MLA attention acceleration + - [FlashInfer](https://github.com/flashinfer-ai/flashinfer) — High-performance inference kernels +- **Communication size modeling**: Supports communication size calculation for TP, DP, PP, EP parallelism strategies +- **Model support**: DeepSeek-V3-671B, Qwen3-MoE-235B, Qwen3-Next-80B + +--- + +## 7. Four-Scenario End-to-End Test Suite + +**File path**: `vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh` + +Provides 4 pre-configured end-to-end test scenarios covering different models, parallelism strategies, and PD disaggregation configurations. + +### Shared Hardware Configuration + +- GPU: H20 (h20_dgx) +- NVLink bandwidth: 1600 Gbps +- RDMA bandwidth: 800 Gbps +- PD P2P bandwidth: 800 Gbps +- Data type: fp8 +- Requests: Poisson QPS=100, 4 requests, fixed prefill=100 / decode=8 tokens + +### Scenario Configuration + +| Scenario | Model | PD Separation | World Size | TP | PP | EP | Global Scheduler | +|----------|-------|---------------|-----------|----|----|-----|-----------------| +| 1 | Qwen3-Next-80B (MoE) | No | 32 (dp=32) | 1 | 1 | 1 | lor | +| 2 | Qwen3-Next-80B (MoE) | Yes (P=2, D=6) | 8 | 1 | 1 | 1 | split_wise | +| 3 | DeepSeek-671B (MoE) | Yes (P=2, D=6) | 8 | 8 | 1 | 8 | split_wise | +| 4 | Qwen3-MoE-235B (MoE) | Yes (P=2, D=6) | 8 | 4 | 1 | 4 | split_wise | + +### Running + +```bash +# Run all 4 scenarios +bash vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh --all + +# Run a single scenario +bash vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh --scenario 3 +``` + +For detailed performance data, please run the test suite. Each run produces output files including `request_metrics.csv` (per-request metrics), `chrome_trace.json` (timeline trace), `config.json` (configuration snapshot), and metric files under the `plots/` directory. + +--- + +## 8. Code Quality Improvements + +SimAI 1.6 includes systematic code quality improvements: + +### 8.1 Bilingual Comments and Documentation + +- Added bilingual (Chinese/English) docstrings to all public APIs +- Added bilingual comments to config, scheduler, predictor, and utils modules +- Added bilingual comments to entity modules +- Shell script outputs and Python runtime outputs use bilingual format + +### 8.2 Logging System Improvements + +- Comprehensive replacement of `print` statements with the `logging` module (~12 files) +- Unified log format using parenthetical bilingual style (e.g., `"GPU总内存 (Total GPU mem): 96.00 GB"`) + +### 8.3 Dead Code Cleanup + +- Removed approximately 390 lines of dead code blocks +- Cleaned up personal debug markers + +### 8.4 TODO Standardization + +- Unified to `TODO(author): description` format +- Added missing type annotations + +--- + +## 9. System Architecture + +### Inference Simulation Data Flow + +``` +Request Generator + | Generate synthetic / real-trace requests + v +Global Scheduler + | Dispatch requests to Prefill / Decode replicas + v +Replica Scheduler + | Batch assembly and scheduling + v +Memory Management (MemoryPlanner + Replica) + | KV cache allocation and capacity checking + v +Execution Time Predictor + | AICB / SimAI Simulation / SimAI Analytical / Vidur + v +Metrics Store + | TTFT, TBT, E2E, communication / compute cost + v +Output (request_metrics.csv, chrome_trace.json, plots/) +``` + +--- + +## 10. Quick Start + +### Environment Setup + +#### Option 1: Docker (Recommended) + +```bash +# Build from project root +docker build -t simai:latest . +docker run --gpus all -it --rm simai:latest +``` + +> If using Hopper GPUs, add `ENV FLASH_MLA_DISABLE_SM100=1` to the Dockerfile. + +#### Option 2: Conda + +```bash +cd vidur-alibabacloud +conda env create -p ./env -f ./environment.yml +conda activate vidur +pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ +``` + +### Run 4-Scenario Test Suite + +```bash +# Prerequisites: conda activate vidur +bash vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh --all +``` + +### Compile and Run SimAI Training Simulation + +```bash +# Compile SimAI-Analytical +./scripts/build.sh -c analytical + +# Run +./bin/SimAI_analytical -w example/workload_analytical.txt -g 9216 -g_p_s 8 -r test- -busbw example/busbw.yaml +``` + +--- + +## 11. References + +[1] SimAI: Unifying Architecture Design and Performance Tuning for Large-Scale Large Language Model Training with Scalability and Precision. NSDI'25 Spring. [[pdf](https://ennanzhai.github.io/pub/nsdi25spring-simai.pdf)] + +[2] InferSim — Alibaba. Parameter counting and KV cache estimation. [[GitHub](https://github.com/alibaba/InferSim)] + +[3] DeepSeek V3 Parameter Derivation (Chinese). Zhihu. [[link](https://zhuanlan.zhihu.com/p/21455638257)] + +[4] DeepSeek V3 Parameter Size Analysis. Yang Wenbo. [[link](https://yangwenbo.com/articles/deepseek-v3-parameter-size.html)] + +[5] Vidur: A Large-Scale Simulation Framework For LLM Inference. Microsoft Research. [[GitHub](https://github.com/microsoft/vidur)] + +[6] splitwise-sim — Prefill-Decode Disaggregation Simulation. [[GitHub](https://github.com/Mutinifni/splitwise-sim)] diff --git a/docs/SimAI_1.6_Tech_Report_CN.md b/docs/SimAI_1.6_Tech_Report_CN.md new file mode 100644 index 00000000..228ae90a --- /dev/null +++ b/docs/SimAI_1.6_Tech_Report_CN.md @@ -0,0 +1,429 @@ ++ 中文 ｜ English +
+ +# SimAI 1.6 技术报告 + +> 本报告涵盖 SimAI 1.5 全部功能及 SimAI 1.6 新增特性。 + +## 1. 概述 + +**SimAI** 是业界首个全栈高精度 AI 大规模推理与训练模拟器（**Sim**ulator for **AI**），由阿里云开源。SimAI 对 LLM 推理与训练全流程进行详细建模和仿真，涵盖框架层、集合通信层、网络传输层等，提供端到端的性能数据。SimAI 论文已被 NSDI'25 Spring 接收 [1]。 + +SimAI 1.6 在 SimAI 1.5 的基础上进一步增强，主要新增了 **GPU 显存计算模块**（支持 DeepSeek-V3-671B、Qwen3-MoE-235B、Qwen3-Next-80B 三种 MoE 模型的精确参数量计算与 KV Cache 管理）、**四场景端到端测试套件**，以及全面的代码质量改进（双语文档、日志系统、死代码清理等）。 + +### 组件构成 + +``` + |--- AICB (工作负载生成与计算 profiling) +SimAI --|--- SimCCL (集合通信算法分析) + |--- astra-sim-alibabacloud (仿真引擎：Analytical / Simulation / Physical) + |--- ns-3-alibabacloud (NS-3 网络后端) + |--- vidur-alibabacloud (多请求推理调度与显存管理) +``` + +--- + +## 2. 关键里程碑 + +以下为 2025 年 11 月至 2026 年 3 月的关键开发事件： + +| 时间 | 事件 | 说明 | +|------|------|------| +| 2025/11 | AICB PR [#58](https://github.com/aliyun/aicb/pull/58) | AICB 新增推理工作负载生成能力，区分 prefill/decode 阶段，支持 DeepSeek、Qwen3-MoE、Qwen3-Next | +| 2025/12 | AICB PR [#60](https://github.com/aliyun/aicb/pull/60) | AICB 进一步更新，完善推理工作负载生成 | +| 2025/12 | SimAI PR [#203](https://github.com/aliyun/SimAI/pull/203) | SimAI 1.5 核心更新：端到端推理仿真、PD 分离、Vidur 调度集成、现代模型支持 | +| 2025/12 | ns-3 commit [7e3cb5b](https://github.com/aliyun/ns-3-alibabacloud/commit/7e3cb5b88c99abcb582c5abc3919484a4805111b) | ns-3-alibabacloud README 文档增强，详细说明 NS3 网络后端修改 | +| 2026/01 | 显存模块系列 commit | 完成 DeepSeek-V3-671B、Qwen3-Next-80B、Qwen3-MoE-235B 的精确显存计算 | +| 2026/02 | PD 分离显存规划 | 实现 Prefill/Decode 阶段独立的参数显存与 KV Cache 预算计算 | +| 2026/03 | 代码质量改进 | 双语注释/文档/日志全面改进，死代码清理，TODO 标准化，类型注解补充 | + +--- + +## 3. 端到端推理仿真 + +SimAI 支持完整的多请求 LLM 推理仿真，核心特性如下： + +### 3.1 Prefill–Decode（PD）分离架构 + +推理过程分为两个阶段： + +- **Prefill 阶段**：处理输入 prompt 的全部 token，生成第一个输出 token（计算密集型） +- **Decode 阶段**：逐 token 自回归生成后续输出（访存密集型） + +PD 分离允许将 Prefill 和 Decode 阶段部署在不同的 GPU 节点上，实现： +- 弹性资源分配（Prefill 节点可配置更多计算资源，Decode 节点可配置更多显存） +- 性能隔离（避免 Prefill 和 Decode 之间的资源争用） +- 灵活的 P:D 节点比例配置（通过 `--replica_config_pd_node_ratio` 控制） + +该设计参考了 [splitwise-sim](https://github.com/Mutinifni/splitwise-sim) [6]。 + +### 3.2 多请求推理调度 + +请求调度组件基于微软 [Vidur](https://github.com/microsoft/vidur) [5] 改编（vidur-alibabacloud），支持以下调度策略： + +| 调度器类型 | 级别 | 说明 | +|-----------|------|------| +| `split_wise` | 全局 | PD 分离场景下的全局调度，将请求分配到 Prefill 和 Decode 副本 | +| `lor` | 全局 | Least Outstanding Requests，将请求分配到负载最轻的副本 | +| `round_robin` | 全局 | 轮询分配 | +| `sarathi` | 副本级 | 单副本内的批处理调度 | +| `split_wise` | 副本级 | PD 分离场景下的副本级调度 | + +### 3.3 灵活的并行策略 + +支持多种并行策略的组合： + +- **数据并行（DP）** — 通过 `--cluster_config_num_replicas` 控制 +- **张量并行（TP）** — 通过 `--replica_config_tensor_parallel_size` 控制 +- **流水线并行（PP）** — 通过 `--replica_config_num_pipeline_stages` 控制 +- **专家并行（EP）** — 通过 `--replica_config_expert_model_parallel_size` 控制 + +同时支持 Dense 模型和 MoE（混合专家）模型。 + +### 3.4 多种执行时间预测后端 + +| 后端 | 说明 | +|------|------| +| **AICB/AIOB** | 部分支持 DeepSeek-V3-671B、Qwen3-MoE-235B、Qwen3-Next-80B 的计算核与 TP/DP/PP/EP 通信量建模 | +| **SimAI Simulation** | 基于 SimAI NS-3 的网络通信全栈仿真（当前支持 TP） | +| **SimAI Analytical** | SimAI 解析性能模型（当前支持 TP） | +| **Native Vidur** | 原版 Vidur 后端，支持 TP、DP、PP | + +--- + +## 4. 现代模型支持 + +SimAI 1.6 支持以下三种前沿 MoE 大模型，模型配置文件位于 `vidur-alibabacloud/data/hf_configs/`： + +### 4.1 DeepSeek-V3-671B + +| 属性 | 值 | +|------|-----| +| 总层数 | 61 | +| 注意力类型 | MLA（Multi-head Latent Attention） | +| 注意力头数 | 128 | +| 隐藏维度 | 7168 | +| KV LoRA 秩 | 512 | +| Q LoRA 秩 | 1536 | +| QK RoPE 头维度 | 64 | +| QK NoPE 头维度 | 128 | +| V 头维度 | 128 | +| MoE 路由专家数 | 256 | +| 每 token 激活专家数 | 8 | +| 共享专家数 | 1 | +| Dense 层（前 3 层） | 固定激活 8 个路由专家 + 1 个共享专家 | +| Sparse 层（第 3-60 层） | 从 256 个路由专家中动态选择 8 个 + 1 个共享专家 | + +配置文件：`data/hf_configs/deepseek_v3_config.json` + +### 4.2 Qwen3-MoE-235B + +| 属性 | 值 | +|------|-----| +| 总层数 | 94 | +| 注意力类型 | MHA/GQA | +| 注意力头数 | 64 | +| KV 头数 | 4 | +| 隐藏维度 | 4096 | +| 头维度 | 128 | +| MoE 路由专家数 | 128 | +| 每 token 激活专家数 | 8 | +| MoE 中间维度 | 1536 | + +配置文件：`data/hf_configs/qwen3_moe_config.json` + +### 4.3 Qwen3-Next-80B + +| 属性 | 值 | +|------|-----| +| 总层数 | 48 | +| 注意力类型 | 混合（全注意力 + 线性注意力，每 4 层交替） | +| 全注意力头数 | 16 | +| KV 头数 | 2 | +| 隐藏维度 | 2048 | +| 头维度 | 256 | +| 线性注意力键头数 | 16 | +| 线性注意力值头数 | 32 | +| MoE 路由专家数 | 512 | +| 每 token 激活专家数 | 10 | +| MoE 中间维度 | 512 | + +配置文件：`data/hf_configs/qwen3-next-80B-A3B_config.json` + +--- + +## 5. GPU 显存计算模块 + +这是 SimAI 1.6 的核心新增特性。该模块为推理仿真提供精确的 GPU 显存估算，覆盖模型参数显存、KV Cache 显存和最大批处理量计算，并在 PD 分离架构下分别计算 Prefill 和 Decode 阶段的显存预算。 + +### 5.1 参数量计算（ParamCounter） + +**文件路径**：`vidur-alibabacloud/vidur/utils/param_counter.py` + +ParamCounter 支持按层、按设备计算模型参数量，并在 PD 分离架构下返回三元组 `(total_params, prefill_params, decode_params)`。 + +#### MLA 参数量（DeepSeek-V3-671B） + +单层 MLA 参数量由以下部分组成： + +- **Q LoRA 下投影**：`wq_down = hidden_size * q_lora_rank` = 7168 * 1536 +- **Q LoRA 上投影**：`wq_up = q_lora_rank * num_attention_heads * qk_head_dim` = 1536 * 128 * 192，其中 `qk_head_dim = qk_nope_head_dim + qk_rope_head_dim = 128 + 64 = 192` +- **KV LoRA 下投影**：`wkv_down = hidden_size * kv_lora_rank` = 7168 * 512 +- **KV LoRA 上投影**：`wkv_up = kv_lora_rank * num_attention_heads * (qk_nope_head_dim + v_head_dim)` = 512 * 128 * 256 +- **输出投影**：`wo = hidden_size * num_attention_heads * v_head_dim` = 7168 * 128 * 128 + +FP8 量化下每个参数元素占 1 字节；FP16/BF16 下每个占 2 字节。 + +参考：[3] [4] + +#### MHA/GQA 参数量（Qwen3-MoE-235B） + +单层 MHA 参数量： + +``` +wq = hidden_size * num_attention_heads * head_dim +wk = hidden_size * num_key_value_heads * head_dim +wv = hidden_size * num_key_value_heads * head_dim +wo = hidden_size * num_attention_heads * head_dim +total = (wq + wk + wv + wo) * bytes_per_element +``` + +#### 线性注意力参数量（Qwen3-Next-80B） + +Qwen3-Next-80B 采用混合注意力架构，每 4 层交替使用全注意力和线性（GDN）注意力。线性注意力层使用独立的键/值头配置（`linear_key_head_dim`、`linear_num_key_heads` 等）。 + +#### MoE 专家参数量 + +每个专家的 FFN 参数量（3 个权重矩阵 W1、W2、W3）： + +``` +expert_params = 3 * hidden_size * moe_intermediate_size * bytes_per_element +``` + +#### PD 分离下的参数量计算 + +在 PD 分离架构下，Prefill 和 Decode 集群的专家并行度（EP）可能不同： + +- **Prefill 集群**：使用 `prefill_world_size` 作为 EP，每设备加载的专家数 = `num_routed_experts / prefill_world_size` +- **Decode 集群**：使用 `decode_world_size` 作为 EP，每设备加载的专家数 = `num_routed_experts / decode_world_size` + +这导致 Prefill 和 Decode 集群的参数显存不同，进而影响各自可用的 KV Cache 容量。 + +### 5.2 KV Cache 显存管理 + +**文件路径**：`vidur-alibabacloud/vidur/scheduler/utils/memory_planner.py`、`vidur-alibabacloud/vidur/entities/replica.py` + +#### MHA/GQA KV Cache 计算 + +``` +kv_cache_per_token = 2 * num_kv_heads * head_dim * num_layers * bytes_per_element +``` + +其中因子 2 代表 K（Key）和 V（Value）两个缓存。 + +#### MLA KV Cache 计算（DeepSeek-V3-671B） + +MLA 架构使用压缩的 KV 表示。与 MHA 分别存储 K 和 V 缓存不同，MLA 存储一个联合编码 K 和 V 的压缩潜向量（`kv_lora_rank`），外加 RoPE 位置键（`qk_rope_head_dim`）。每 token 的 KV Cache 大小为： + +``` +kv_cache_per_token = (kv_lora_rank + qk_rope_head_dim) * num_layers * bytes_per_element +``` + +其中 `kv_lora_rank = 512`，`qk_rope_head_dim = 64`。相比 MHA 每 token 缓存量 `2 * num_kv_heads * head_dim` = 2 * 128 * 128 = 32768 个元素，MLA 减少至 576 个元素——约 **57 倍**压缩。 + +#### 逐请求 KV Cache 追踪 + +`Replica` 实体（`vidur/entities/replica.py`）维护以下状态： + +- `_allocated_kv_cache_memory`：已分配的 KV Cache 显存（字节） +- `_max_kv_cache_memory`：最大 KV Cache 容量（首次调用时由 MemoryPlanner 计算） +- `_kv_cache_allocation_map`：每请求 KV Cache 分配映射 + +支持的操作： +- `allocate_request_kv_cache_memory(request, num_blocks, block_size)` — 为请求分配 KV Cache +- `release_request_kv_cache_memory(request)` — 释放已完成请求的 KV Cache +- `get_remaining_kv_cache_capacity()` — 查询剩余 KV Cache 容量和可服务请求数 + +### 5.3 MemoryPlanner 显存规划 + +**文件路径**：`vidur-alibabacloud/vidur/scheduler/utils/memory_planner.py` + +MemoryPlanner 是显存管理的核心组件，计算流程如下： + +1. **计算可用 GPU 显存**：`available_memory = total_GPU_memory * (1 - memory_margin_fraction)` +2. **获取模型参数显存**：通过 ParamCounter 计算，PD 分离下返回 `(total, prefill, decode)` 三元组 +3. **计算 KV Cache 可用显存**：`kv_cache_available = available_memory - param_memory` +4. **计算最大并发请求数**：`max_requests = kv_cache_available / kv_cache_per_request` + +在 PD 分离架构下： +- Prefill 副本使用 `prefill_param_mem` 计算 KV Cache 预算 +- Decode 副本使用 `decode_param_mem` 计算 KV Cache 预算 + +包含 OOM 检测：当参数显存超过可用显存时，输出错误信息并建议增加 TP/EP、使用更大 GPU 或启用 FP8 量化。 + +--- + +## 6. AICB 推理工作负载生成 + +[AICB](https://github.com/aliyun/aicb) 新增了推理工作负载生成能力（PR [#58](https://github.com/aliyun/aicb/pull/58)、[#60](https://github.com/aliyun/aicb/pull/60)），主要特性： + +- **Prefill/Decode 阶段分离**：分别生成 Prefill 和 Decode 阶段的计算与通信工作负载 +- **计算核 Profiling**：依赖以下硬件加速库（需要 Hopper SM90 或 Blackwell SM100 GPU）： + - [DeepGEMM](https://github.com/deepseek-ai/DeepGEMM) — FP8 矩阵乘法 + - [FlashMLA](https://github.com/deepseek-ai/FlashMLA) — MLA 注意力加速 + - [FlashInfer](https://github.com/flashinfer-ai/flashinfer) — 高性能推理内核 +- **通信量建模**：支持 TP、DP、PP、EP 四种并行策略下的通信量计算 +- **模型支持**：DeepSeek-V3-671B、Qwen3-MoE-235B、Qwen3-Next-80B + +--- + +## 7. 四场景端到端测试套件 + +**文件路径**：`vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh` + +提供了 4 个预配置的端到端测试场景，覆盖不同模型、并行策略和 PD 分离配置。 + +### 共用硬件配置 + +- GPU：H20 (h20_dgx) +- NVLink 带宽：1600 Gbps +- RDMA 带宽：800 Gbps +- PD P2P 带宽：800 Gbps +- 数据类型：fp8 +- 请求：Poisson QPS=100，4 个请求，固定 prefill=100 / decode=8 tokens + +### 场景配置表 + +| 场景 | 模型 | PD 分离 | World Size | TP | PP | EP | 全局调度 | +|------|------|---------|-----------|----|----|-----|---------| +| 1 | Qwen3-Next-80B (MoE) | 否 | 32 (dp=32) | 1 | 1 | 1 | lor | +| 2 | Qwen3-Next-80B (MoE) | 是 (P=2, D=6) | 8 | 1 | 1 | 1 | split_wise | +| 3 | DeepSeek-671B (MoE) | 是 (P=2, D=6) | 8 | 8 | 1 | 8 | split_wise | +| 4 | Qwen3-MoE-235B (MoE) | 是 (P=2, D=6) | 8 | 4 | 1 | 4 | split_wise | + +### 运行方式 + +```bash +# 运行全部 4 个场景 +bash vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh --all + +# 运行单个场景 +bash vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh --scenario 3 +``` + +详细性能数据请运行测试套件获取。每次运行产生的输出文件包括 `request_metrics.csv`（逐请求指标）、`chrome_trace.json`（时间线追踪）、`config.json`（配置快照）以及 `plots/` 目录下的指标文件。 + +--- + +## 8. 代码质量改进 + +SimAI 1.6 在代码质量方面进行了系统性改进： + +### 8.1 双语注释与文档 + +- 所有公共 API 添加中英双语 docstring +- 配置模块（config）、调度器（scheduler）、预测器（predictor）、工具类（utils）添加双语注释 +- 实体模块（entities）添加双语注释 +- Shell 脚本输出和 Python 运行时输出均为双语格式 + +### 8.2 日志系统改进 + +- 全面将 `print` 语句替换为 `logging` 模块（涉及 ~12 个文件） +- 统一日志格式，使用括号式双语格式（如 `"GPU总内存 (Total GPU mem): 96.00 GB"`） + +### 8.3 死代码清理 + +- 移除约 390 行无效代码块 +- 清理个人调试标记 + +### 8.4 TODO 标准化 + +- 统一为 `TODO(author): description` 格式 +- 补充缺失的类型注解 + +--- + +## 9. 系统架构 + +### 推理仿真数据流 + +``` +请求生成器 (Request Generator) + │ 生成合成/真实 trace 请求 + ▼ +全局调度器 (Global Scheduler) + │ 将请求分配到 Prefill/Decode 副本 + ▼ +副本调度器 (Replica Scheduler) + │ 批处理组装与调度 + ▼ +显存管理 (MemoryPlanner + Replica) + │ KV Cache 分配与容量检查 + ▼ +执行时间预测 (Execution Time Predictor) + │ AICB / SimAI Simulation / SimAI Analytical / Vidur + ▼ +指标收集 (Metrics Store) + │ TTFT, TBT, E2E, 通信/计算开销 + ▼ +输出 (request_metrics.csv, chrome_trace.json, plots/) +``` + +--- + +## 10. 快速开始 + +### 环境搭建 + +#### 方式一：Docker（推荐） + +```bash +# 从项目根目录构建 +docker build -t simai:latest . +docker run --gpus all -it --rm simai:latest +``` + +> 若使用 Hopper GPU，请在 Dockerfile 中添加 `ENV FLASH_MLA_DISABLE_SM100=1`。 + +#### 方式二：Conda + +```bash +cd vidur-alibabacloud +conda env create -p ./env -f ./environment.yml +conda activate vidur +pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ +``` + +### 运行四场景测试 + +```bash +# 前置条件：conda activate vidur +bash vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh --all +``` + +### 编译与运行 SimAI 训练仿真 + +```bash +# 编译 SimAI-Analytical +./scripts/build.sh -c analytical + +# 运行 +./bin/SimAI_analytical -w example/workload_analytical.txt -g 9216 -g_p_s 8 -r test- -busbw example/busbw.yaml +``` + +--- + +## 11. 参考文献 + +[1] SimAI: Unifying Architecture Design and Performance Tuning for Large-Scale Large Language Model Training with Scalability and Precision. NSDI'25 Spring. [[pdf](https://ennanzhai.github.io/pub/nsdi25spring-simai.pdf)] + +[2] InferSim — Alibaba. Parameter counting and KV cache estimation. [[GitHub](https://github.com/alibaba/InferSim)] + +[3] DeepSeek V3 参数推导详解. 知乎. [[link](https://zhuanlan.zhihu.com/p/21455638257)] + +[4] DeepSeek V3 Parameter Size Analysis. Yang Wenbo. [[link](https://yangwenbo.com/articles/deepseek-v3-parameter-size.html)] + +[5] Vidur: A Large-Scale Simulation Framework For LLM Inference. Microsoft Research. [[GitHub](https://github.com/microsoft/vidur)] + +[6] splitwise-sim — Prefill-Decode Disaggregation Simulation. [[GitHub](https://github.com/Mutinifni/splitwise-sim)] diff --git a/docs/en/benchmarking/index.md b/docs/en/benchmarking/index.md new file mode 100644 index 00000000..2431cf17 --- /dev/null +++ b/docs/en/benchmarking/index.md @@ -0,0 +1,33 @@ +# Benchmarking + +This section covers benchmarking and validation approaches for SimAI. + +--- + +## Contents + +| Document | Description | +|----------|-------------| +| [4-Scenario End-to-End Test Suite](test_suite.md) | Pre-configured test scenarios covering different models, parallelism strategies, and PD configurations | + +--- + +## Benchmarking Approaches + +SimAI supports several benchmarking methodologies: + +### Architecture Comparison + +Compare different network architectures (e.g., Spectrum-X vs DCN+) under identical workloads to evaluate their performance characteristics. + +### Algorithm Comparison + +Compare different collective communication algorithms (e.g., RING vs NVLS) to understand their performance trade-offs at various message sizes. + +### Parameter Optimization + +Use SimAI-Analytical for rapid exploration of parallel parameter combinations (TP, PP, EP, DP) to find optimal configurations. + +### Validation Against Real Hardware + +Use AICB physical execution results as ground truth to validate simulation accuracy. diff --git a/docs/en/benchmarking/test_suite.md b/docs/en/benchmarking/test_suite.md new file mode 100644 index 00000000..54a54404 --- /dev/null +++ b/docs/en/benchmarking/test_suite.md @@ -0,0 +1,139 @@ +# 4-Scenario End-to-End Test Suite + +SimAI provides a pre-configured test suite covering 4 representative inference scenarios, enabling quick validation of all supported configurations. + +--- + +## Overview + +The test suite is located at `vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh` and covers different combinations of models, parallelism strategies, and PD disaggregation configurations. + +--- + +## Running + +```bash +# Run all 4 scenarios +bash vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh --all + +# Run a single scenario +bash vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh --scenario 1 + +# Show help +bash vidur-alibabacloud/examples/vidur-ali-scenarios/run_scenarios.sh --help +``` + +> **Prerequisites**: `conda activate vidur` environment must be active. + +--- + +## Shared Hardware Configuration + +All scenarios share the following hardware settings: + +| Parameter | Value | +|-----------|-------| +| GPU | H20 (h20_dgx) | +| NVLink Bandwidth | 1600 Gbps | +| RDMA Bandwidth | 800 Gbps | +| PD P2P Bandwidth | 800 Gbps | +| PD P2P Data Type | fp8 | +| Request Generator | Poisson, QPS=100 | +| Request Count | 4 | +| Prefill Tokens | 100 (fixed) | +| Decode Tokens | 8 (fixed) | + +--- + +## Scenario Configuration + +| Scenario | Model | PD Separation | World Size | TP | PP | EP | Global Scheduler | +|----------|-------|---------------|-----------|----|----|-----|-----------------| +| **1** | Qwen3-Next-80B (MoE) | No | 32 (dp=32) | 1 | 1 | 1 (default) | lor | +| **2** | Qwen3-Next-80B (MoE) | Yes (P=2, D=6) | 8 | 1 | 1 | 1 (default) | split_wise | +| **3** | DeepSeek-671B (MoE) | Yes (P=2, D=6) | 8 | 8 | 1 | 8 | split_wise | +| **4** | Qwen3-MoE-235B (MoE) | Yes (P=2, D=6) | 8 | 4 | 1 | 4 | split_wise | + +### Scenario Details + +- **Scenario 1**: Large-scale DP without PD separation — tests baseline throughput +- **Scenario 2**: Same model with PD separation — tests PD disaggregation overhead +- **Scenario 3**: DeepSeek-671B with large TP/EP — tests MoE with MLA attention +- **Scenario 4**: Qwen3-MoE-235B with moderate TP/EP — tests MHA/GQA attention model + +--- + +## Output + +### Output Directory + +- **Via run_scenarios.sh**: `examples/vidur-ali-scenarios/simulator_output/` +- **Direct python**: `./simulator_output/` + +### Output Files + +``` +
+ 
+ 
+
+
+---
+
+## Events
+
+### Upcoming Events
+
+| Date | Event | Location | Type |
+|:----:|:------|:---------|:----:|
+| — | — | — | — |
+
+### Past Events
+
+| Date | Event | Location | Type |
+|:----:|:------|:---------|:----:|
+| Dec 30, 2025 | SimAI 1.5 Release | Online | Virtual |
+| Jun 4, 2025 | First SimAI Community Workshop | Peking University | On-site |
+| May 24, 2025 | 28th Chinasys Workshop — SimAI Talk | Chongqing University | On-site |
+| Dec 27, 2024 | SimAI Technical Presentation | Beihang University | On-site |
+| Dec 6, 2024 | HKUST Technical Workshop | HKUST(GZ) | On-site |
+| Dec 5, 2024 | Bench'24 Conference — SimAI Tutorial | Guangzhou | On-site |
+| Nov 26, 2024 | SimAI Community Live Stream (400+ attendees) | Online | Virtual |
+| Nov 15, 2024 | Technical Workshop | Thousand Island Lake | On-site |
+| Oct 18, 2024 | Guest Lecture — SimAI Tutorial | Fudan University | On-site |
+| Sept 24-26, 2024 | CCF HPC China 2024 | Wuhan | Conference |
+
+---
+
+## Citation
+
+SimAI has been accepted by **NSDI'25 Spring**. If you use SimAI in your research, please cite:
+
+```bibtex
+@inproceedings{simai-nsdi25,
+ title={SimAI: Unifying Architecture Design and Performance Tuning for Large-Scale Large Language Model Training with Scalability and Precision},
+ booktitle={NSDI'25 Spring},
+ year={2025}
+}
+```
+
+**Resources:**
+- [Paper (PDF)](https://ennanzhai.github.io/pub/nsdi25spring-simai.pdf)
+- [Slides](../../docs/SimAI_Intro_Online.pdf)
+- [Video](https://n.dingtalk.com/dingding/live-room/index.html?roomId=OF5BkBUXVxmgsK7x&liveUuid=305736cd-aa70-498b-8003-2b471a53decd)
+
+---
+
+## Acknowledgments
+
+Major contributors:
+
+- TianHao Fu (Peking University) and [TELOS-syslab](https://github.com/TELOS-syslab/)
+- Parth Parikh (KEYSIGHT)
+- Sarah-Michelle Hammer & Ziyi Wang (TU-Berlin)
+- Xinyue Li (BUPT)
+- Tong Chen (Zhejiang University)
+- Ming Wang (BUPT)
+- Tao Jiang (Institute of Computing Technology, Chinese Academy of Sciences)
+
+And many other individual contributors — see [Contributors to aliyun/SimAI](https://github.com/aliyun/SimAI/graphs/contributors).
+
+Special thanks to Chenning Li (MIT CSAIL) for initiating the cooperation on integrating SimAI into [M4](https://github.com/netiken/m4).
+
+---
+
+## Contributing
+
+We warmly welcome contributions! See our [Contributing Guide](../developer_guide/contributing.md) for details.
diff --git a/docs/en/components/aicb.md b/docs/en/components/aicb.md
new file mode 100644
index 00000000..b6765545
--- /dev/null
+++ b/docs/en/components/aicb.md
@@ -0,0 +1,212 @@
+# AICB — AI Communication Benchmark
+
+**Repository**: [aliyun/aicb](https://github.com/aliyun/aicb) | **Language**: Python
+
+AICB is a specialized communication benchmarking suite for AI scenarios. It generates realistic communication workloads aligned to real-world LLM training and inference processes.
+
+---
+
+## Introduction
+
+AICB (Artificial Intelligence Communication Benchmark) produces communication workloads with precise patterns aligned to real-world applications. It supports:
+
+- Benchmarking and tuning GPU cluster communication systems
+- Investigating communication patterns of specific model configurations
+- Generating workloads for simulators like SimAI
+
+---
+
+## Benchmark Suite
+
+AICB provides 10 pre-configured benchmark cases covering typical LLM configurations:
+
+| ID | Model | Seq Length | Framework | TP | PP | SP | MoE |
+|----|-------|-----------|-----------|----|----|-----|-----|
+| 1 | LLaMA-7B | 2048 | Megatron | 1 | 1 | - | - |
+| 2 | GPT-13B | 2048 | Megatron | 2 | 1 | Yes | - |
+| 3 | GPT-22B | 2048 | Megatron | 4 | 1 | - | - |
+| 4 | LLaMA-65B | 4096 | Megatron | 8 | 2 | Yes | - |
+| 5 | GPT-175B | 2048 | Megatron | 8 | 8 | Yes | - |
+| 6 | GPT-175B | 2048 | Megatron | 8 | 8 | - | - |
+| 7 | Llama3-405B | 8192 | Megatron | 8 | 16 | Yes | - |
+| 8 | LLaMA-7B | 4096 | DeepSpeed | 1 | 1 | - | Zero-2 |
+| 9 | LLaMA-65B | 4096 | DeepSpeed | 1 | 1 | - | Zero-3 |
+| 10 | Mistral-8x7B | 2048 | Megatron | 2 | 1 | Yes | 8 experts |
+
+---
+
+## Environment Setup
+
+### Docker
+
+```bash
+docker build -t aicb:v0.0.1 .
+docker run --gpus all --net host --shm-size 16g -it --rm aicb:v0.0.1
+```
+
+### Local Environment
+
+Requirements: Python >= 3.8, CUDA >= 11.8, PyTorch >= 2.0.0, NVIDIA APEX
+
+### NGC Container
+
+```bash
+docker pull nvcr.io/nvidia/pytorch:xx.xx-py3
+docker run --gpus all -it --rm -v /path/to/aicb:/workspace/aicb nvcr.io/nvidia/pytorch:xx.xx-py3
+```
+
+> **Note**: Inference workload profiling requires NVIDIA Hopper (SM90) or Blackwell (SM100) GPUs.
+
+---
+
+## Physical Execution on GPU Clusters
+
+### Environment Variables
+
+| Parameter | Description |
+|-----------|-------------|
+| `nnodes` | Number of nodes |
+| `node_rank` | Rank of the node |
+| `nproc_per_node` | Number of GPUs per node |
+| `master_addr` | Master node address |
+| `master_port` | Master node port |
+
+### Running Megatron Workloads
+
+```bash
+sh scripts/megatron_gpt.sh \
+ --nnodes 1 --node_rank 0 --nproc_per_node 8 \
+ --master_addr localhost --master_port 29500 \
+ -m 7 --world_size 8 --tensor_model_parallel_size 2 --pipeline_model_parallel 1 \
+ --frame Megatron --global_batch 16 --micro_batch 1 \
+ --seq_length 2048 --swiglu --use_flash_attn --aiob_enable
+```
+
+### Running MoE Workloads
+
+```bash
+sh scripts/megatron_gpt.sh \
+ -m moe --world_size 8 --tensor_model_parallel_size 4 \
+ --moe_enable --expert_model_parallel_size 1 \
+ --num_experts 4 --moe_router_topk 2 \
+ --frame Megatron --global_batch 16 --micro_batch 1 \
+ --sp --grouped_gemm --aiob_enable --swiglu --use_flash_attn
+```
+
+### Running DeepSeek Workloads
+
+```bash
+sh scripts/megatron_gpt.sh \
+ --frame DeepSeek -m deepseek \
+ --tensor_model_parallel_size 4 --moe_enable \
+ --expert_model_parallel_size 1 --num_experts 4 \
+ --global_batch 4 --micro_batch 1 --world_size 4 \
+ --num_layers 10 --sp --swiglu --aiob_enable
+```
+
+---
+
+## Training Workload Generation
+
+Generate workload files for SimAI simulation:
+
+```bash
+python -m workload_generator.SimAI_training_workload_generator \
+ --model_name GPT-13B --frame=Megatron \
+ --world_size=16 --tensor_model_parallel_size=2 --pipeline_model_parallel=1 \
+ --global_batch=16 --micro_batch=1 --num_layers=40 --seq_length=2048 \
+ --hidden_size=5120 --epoch_num=1 --num_attention_heads=40 \
+ --aiob_enable --use_flash_attn --swiglu
+```
+
+Output saved in `results/mocked_workload/`.
+
+---
+
+## Inference Workload Generation
+
+AICB generates inference workloads with prefill/decode phase separation for:
+
+| Model | Attention | MoE Experts |
+|-------|-----------|-------------|
+| DeepSeek-V3-671B | MLA | 256 routed + 1 shared |
+| Qwen3-MoE-235B | MHA/GQA | 128 routed |
+| Qwen3-Next-80B | Hybrid (full + linear) | 512 routed |
+
+Requires hardware-accelerated libraries: [DeepGEMM](https://github.com/deepseek-ai/DeepGEMM), [FlashMLA](https://github.com/deepseek-ai/FlashMLA), [FlashInfer](https://github.com/flashinfer-ai/flashinfer).
+
+---
+
+## AIOB: Computation Profiling
+
+AIOB profiles actual GPU computation times and embeds them into workloads:
+
+- `--aiob_enable` — Profile on current GPU
+- `--comp_filepath
+
++ 中文 ｜ English +
+ +[](../../LICENSE) +[](https://ennanzhai.github.io/pub/nsdi25spring-simai.pdf) + +**SimAI** is the industry's first full-stack, high-precision **Sim**ulator for **AI** large-scale **inference** and **training**, open-sourced by Alibaba Cloud. It provides detailed modeling and simulation of the entire LLM training and inference process, encompassing the framework layer, collective communication layer, and network transport layer, delivering end-to-end performance data. + +SimAI enables researchers to: + +- Analyze inference/training process details +- Evaluate the time consumption of AI tasks under specific conditions +- Evaluate E2E performance gains from various algorithmic optimizations (framework parameters, collective communication algorithms, network protocols, congestion control, routing, topology, etc.) + +--- + +## Documentation Overview + +| Section | Description | +|---------|-------------| +| [Getting Started](getting_started/index.md) | Installation, environment setup, and quickstart guide | +| [User Guide](user_guide/index.md) | Detailed usage for SimAI-Analytical, SimAI-Simulation, SimAI-Physical, and Inference Simulation | +| [Components](components/index.md) | In-depth documentation for each submodule: AICB, SimCCL, astra-sim, ns-3, vidur | +| [Technical Reference](technical_reference/index.md) | GPU memory module, CLI parameters, and configuration reference | +| [Benchmarking](benchmarking/index.md) | 4-scenario end-to-end test suite and benchmark results | +| [Developer Guide](developer_guide/index.md) | Architecture, contributing guide, adding models, and extending NS-3 | +| [Community](community/index.md) | Events, contact information, and citation | + +--- + +## Architecture + +``` + |--- AICB (Workload generation & compute profiling) +SimAI --|--- SimCCL (Collective communication algorithm analysis) + |--- astra-sim-alibabacloud (Simulation engine: Analytical / Simulation / Physical) + |--- ns-3-alibabacloud (NS-3 network backend) + |--- vidur-alibabacloud (Multi-request inference scheduling & memory management) +``` + + + +--- + +## Three Operation Modes + +| Mode | Description | Use Cases | +|------|-------------|-----------| +| **SimAI-Analytical** | Fast simulation using bus bandwidth (busbw) to estimate collective communication time | Performance analysis, parallel parameter optimization, scale-up exploration | +| **SimAI-Simulation** | Full-stack simulation with NS-3 network backend for fine-grained network modeling | CC algorithm research, network protocol evaluation, novel architecture design | +| **SimAI-Physical** *(Beta)* | Physical traffic generation on CPU RDMA clusters | NIC behavior study during LLM training | + +--- + +## Supported Models + +- **DeepSeek-V3-671B** — MLA attention, 256 routed experts +- **Qwen3-MoE-235B** — MHA/GQA, 128 routed experts +- **Qwen3-Next-80B** — Hybrid full + linear attention, 512 routed experts +- **Meta-Llama-3-8B / 70B**, **Llama-2-7b / 70b**, **CodeLlama-34b**, **InternLM-20B**, **Qwen-72B** + +--- + +## Quick Links + +- [GitHub Repository](https://github.com/aliyun/SimAI) +- [NSDI'25 Paper (PDF)](https://ennanzhai.github.io/pub/nsdi25spring-simai.pdf) +- [Slides](../../docs/SimAI_Intro_Online.pdf) +- [Technical Report (1.6)](../SimAI_1.6_Tech_Report.md) +- [Contributing Guide](../../CONTRIBUTING.md) diff --git a/docs/en/technical_reference/cli_reference.md b/docs/en/technical_reference/cli_reference.md new file mode 100644 index 00000000..689d2058 --- /dev/null +++ b/docs/en/technical_reference/cli_reference.md @@ -0,0 +1,185 @@ +# CLI Reference + +Complete command-line parameter reference for all SimAI tools. + +--- + +## SimAI-Analytical + +**Binary**: `bin/SimAI_analytical` + +### Required Parameters + +| Flag | Long Form | Description | +|------|-----------|-------------| +| `-w` | `--workload` | Path to workload file | +| `-g` | `--gpus` | Simulation GPU scale | +| `-g_p_s` | `--gpus-per-server` | Scale-up size (GPUs per server) | +| `-r` | `--result` | Output file path and prefix (default: `./results/`) | +| `-busbw` | `--bus-bandwidth` | Path to busbw.yaml file | + +### Optional Parameters + +| Flag | Long Form | Description | +|------|-----------|-------------| +| `-v` | `--visual` | Generate visualization files | +| `-dp_o` | `--dp-overlap-ratio` | DP overlap ratio [0.0-1.0] | +| `-ep_o` | `--ep-overlap-ratio` | EP overlap ratio [0.0-1.0] | +| `-tp_o` | `--tp-overlap-ratio` | TP overlap ratio [0.0-1.0] | +| `-pp_o` | `--pp-overlap-ratio` | PP overlap ratio [0.0-1.0] | + +### Auto Busbw Calculation + +| Flag | Description | +|------|-------------| +| `-nv` | NVLink bandwidth (GB/s) | +| `-nic` | NIC bandwidth (GB/s) | +| `-n_p_s` | NICs per server | + +--- + +## SimAI-Simulation + +**Binary**: `bin/SimAI_simulator` + +### Environment Variables + +| Variable | Description | Default | +|----------|-------------|---------| +| `AS_LOG_LEVEL` | Log level: DEBUG/INFO/WARNING/ERROR | `INFO` | +| `AS_PXN_ENABLE` | Enable PXN | `0` | +| `AS_NVLS_ENABLE` | Enable NVLS | `0` | +| `AS_SEND_LAT` | Send latency (us) | `6` | +| `AS_NVLSTREE_ENABLE` | Enable NVLS Tree | `false` | + +### Parameters + +| Flag | Long Form | Description | Default | +|------|-----------|-------------|---------| +| `-t` | `--thread` | Number of threads | `1` | +| `-w` | `--workload` | Path to workload | Required | +| `-n` | `--network-topo` | Topology file path | Required | +| `-c` | `--config` | SimAI.conf path | Required | + +--- + +## SimAI-Physical + +**Binary**: `bin/SimAI_phynet` + +| Parameter | Description | Default | +|-----------|-------------|---------| +| `hostlist` | Path to host IP list | Required | +| `-w` / `--workload` | Workload file path | `./microAllReduce.txt` | +| `-i` / `--gid_index` | GID index for RDMA | `0` | +| `-g` / `--gpus` | Number of GPUs | `8` | + +--- + +## Topology Generator + +**Script**: `astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py` + +| Level | Flag | Description | +|-------|------|-------------| +| Global | `-topo` | Template: Spectrum-X / AlibabaHPN / DCN+ | +| | `-g` | Number of GPUs | +| | `--dp` | Enable dual plane | +| | `--ro` | Enable rail-optimized | +| | `--dt` | Enable dual ToR | +| | `-er` | Error rate | +| Intra-Host | `-gps` | GPUs per server | +| | `-gt` | GPU type (A100/H100) | +| | `-nsps` | NV switches per server | +| | `-nvbw` | NVLink bandwidth | +| | `-nl` | NVLink latency | +| | `-l` | NIC latency | +| Intra-Segment | `-bw` | NIC to ASW bandwidth | +| | `-asw` | ASW switch count | +| | `-nps` | NICs per switch | +| Intra-Pod | `-psn` | PSW switch count | +| | `-apbw` | ASW to PSW bandwidth | +| | `-app` | ASW per PSW | + +--- + +## AICB Workload Generator + +**Script**: `scripts/megatron_workload_with_aiob.sh` or `python -m workload_generator.SimAI_training_workload_generator` + +### Core Parameters + +| Parameter | Description | +|-----------|-------------| +| `--frame` | Framework: Megatron / DeepSpeed / DeepSeek | +| `-m` / `--model_size` | Model size: 7/13/22/175/moe/deepseek | +| `--world_size` | Total GPU count | +| `--global_batch` | Total batch size | +| `--micro_batch` | Micro-batch size | +| `--seq_length` | Sequence length | +| `--epoch_num` | Number of iterations | + +### Parallelism Parameters + +| Parameter | Description | +|-----------|-------------| +| `--tensor_model_parallel_size` | TP degree | +| `--pipeline_model_parallel` | PP degree | +| `--expert_model_parallel_size` | EP degree | +| `--enable_sequence_parallel` | Enable SP | + +### Model Parameters + +| Parameter | Description | +|-----------|-------------| +| `--num_layers` | Transformer layers | +| `--hidden_size` | Hidden size | +| `--num_attention_heads` | Attention heads | +| `--ffn_hidden_size` | FFN hidden size | +| `--vocab_size` | Vocabulary size | + +### MoE Parameters + +| Parameter | Description | +|-----------|-------------| +| `--moe_enable` | Enable MoE | +| `--num_experts` | Number of experts | +| `--moe_router_topk` | Experts per token | +| `--moe_grouped_gemm` | Enable grouped GEMM | + +### DeepSeek Parameters + +| Parameter | Description | +|-----------|-------------| +| `--qk_rope_dim` | RoPE dimension for QK | +| `--qk_nope_dim` | Non-RoPE dimension for QK | +| `--q_lora_rank` | Q LoRA rank | +| `--kv_lora_rank` | KV LoRA rank | +| `--v_head_dim` | V head dimension | +| `--n_shared_expert` | Shared experts per MoE layer | +| `--n_dense_layer` | Dense layers count | + +### Optimization Parameters + +| Parameter | Description | +|-----------|-------------| +| `--use_flash_attn` | FlashAttention | +| `--swiglu` | SwiGLU activation | +| `--aiob_enable` | AIOB computation profiling | +| `--comp_filepath` | Pre-computed times file | + +--- + +## Vidur Inference Simulation + +**Command**: `python -m vidur.main` + +Run `python -m vidur.main -h` for the full parameter list. Key parameters are documented in the [vidur component page](../components/vidur.md). + +--- + +## See Also + +- [Configuration Reference](configuration.md) — Config file formats +- [SimAI-Analytical Guide](../user_guide/simai_analytical.md) — Usage examples +- [AICB Component](../components/aicb.md) — Full parameter details diff --git a/docs/en/technical_reference/configuration.md b/docs/en/technical_reference/configuration.md new file mode 100644 index 00000000..7ad67fb6 --- /dev/null +++ b/docs/en/technical_reference/configuration.md @@ -0,0 +1,163 @@ +# Configuration Reference + +This document covers all configuration files used by SimAI. + +--- + +## SimAI.conf + +**Path**: `astra-sim-alibabacloud/inputs/config/SimAI.conf` + +The main simulation configuration file used by both SimAI-Analytical and SimAI-Simulation modes. It controls communication algorithms, buffer sizes, and timing parameters. + +--- + +## busbw.yaml + +**Path**: `example/busbw.yaml` + +Used by SimAI-Analytical to specify bus bandwidth for different communication groups and collective operations. + +### Format + +```yaml +test +TP: + allreduce,: 300 # AllReduce busbw 300GB/s in TP group + allgather,: 280 + reducescatter,: 280 + alltoall,: 230 +DP: + allreduce,: null # null = not used in this group + allgather,: 380 + reducescatter,: 380 + alltoall,: null +EP: + allreduce,: null + allgather,: 45 + reducescatter,: 45 + alltoall,: 80 +``` + +### Communication Groups + +| Group | Description | +|-------|-------------| +| `TP` | Tensor Parallelism — intra-server NVLink communication | +| `DP` | Data Parallelism — inter-server RDMA communication | +| `EP` | Expert Parallelism — MoE expert communication | + +### Collective Operations + +| Operation | Description | +|-----------|-------------| +| `allreduce` | Reduce + broadcast across all ranks | +| `allgather` | Gather data from all ranks | +| `reducescatter` | Reduce and scatter | +| `alltoall` | All-to-all personalized exchange | + +Set value to `null` for operations not used in a particular group. + +--- + +## Topology Files + +Generated by `gen_Topo_Template.py`, topology files define the network structure for SimAI-Simulation. + +### Generation + +```bash +python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py \ + -topo Spectrum-X -g 128 -gt A100 -bw 100Gbps -nvbw 2400Gbps +``` + +The output file is named based on parameters, e.g., `Spectrum-X_128g_8gps_100Gbps_A100`. + +### Template Defaults + +| Template | GPUs | Topo | Bandwidth | GPU Type | +|----------|------|------|-----------|----------| +| Spectrum-X | 4096 | Rail-optimized, single ToR | 400Gbps | H100 | +| AlibabaHPN (single) | 15360 | Rail-optimized, dual ToR | 200Gbps | H100 | +| AlibabaHPN (dual) | 15360 | Rail-optimized, dual ToR, dual plane | 200Gbps | H100 | +| DCN+ (single ToR) | 512 | Non rail-optimized | 400Gbps | A100 | +| DCN+ (dual ToR) | 512 | Non rail-optimized, dual ToR | 200Gbps | H100 | + +--- + +## Model Configuration Files + +### Inference Model Configs + +Located in `vidur-alibabacloud/data/hf_configs/`: + +| Model | Config File | +|-------|------------| +| DeepSeek-V3-671B | `deepseek_v3_config.json` | +| Qwen3-MoE-235B | `qwen3_moe_config.json` | +| Qwen3-Next-80B | `qwen3-next-80B-A3B_config.json` | + +These files follow the HuggingFace `config.json` format and define model architecture parameters. + +### Profiling Data + +Located in `vidur-alibabacloud/data/profiling/`: + +``` +profiling/ +├── compute/ +│ ├── a100/ +│ │ └──
+
+
+
+
+---
+
+## 活动
+
+### 即将举办
+
+| 日期 | 活动 | 地点 | 类型 |
+|:----:|:------|:---------|:----:|
+| — | — | — | — |
+
+### 历史活动
+
+| 日期 | 活动 | 地点 | 类型 |
+|:----:|:------|:---------|:----:|
+| 2025/12/30 | SimAI 1.5 发布 | 线上 | 线上 |
+| 2025/06/04 | 首届 SimAI 社区研讨会 | 北京大学 | 线下 |
+| 2025/05/24 | 第 28 届 Chinasys Workshop — SimAI 报告 | 重庆大学 | 线下 |
+| 2024/12/27 | SimAI 技术报告 | 北京航空航天大学 | 线下 |
+| 2024/12/06 | 香港科技大学（广州）技术研讨会 | 香港科技大学（广州） | 线下 |
+| 2024/12/05 | Bench'24 会议 — SimAI 教程 | 广州 | 线下 |
+| 2024/11/26 | SimAI 社区直播（400+ 参与者） | 线上 | 线上 |
+| 2024/11/15 | 技术研讨会 | 千岛湖 | 线下 |
+| 2024/10/18 | 客座讲座 — SimAI 教程 | 复旦大学 | 线下 |
+| 2024/09/24-26 | CCF HPC China 2024 | 武汉 | 会议 |
+
+---
+
+## 引用
+
+SimAI 已被 **NSDI'25 Spring** 接收。如果您在研究中使用了 SimAI，请引用：
+
+```bibtex
+@inproceedings{simai-nsdi25,
+ title={SimAI: Unifying Architecture Design and Performance Tuning for Large-Scale Large Language Model Training with Scalability and Precision},
+ booktitle={NSDI'25 Spring},
+ year={2025}
+}
+```
+
+**相关资源：**
+- [论文 (PDF)](https://ennanzhai.github.io/pub/nsdi25spring-simai.pdf)
+- [幻灯片](../../docs/SimAI_Intro_Online.pdf)
+- [视频回放](https://n.dingtalk.com/dingding/live-room/index.html?roomId=OF5BkBUXVxmgsK7x&liveUuid=305736cd-aa70-498b-8003-2b471a53decd)
+
+---
+
+## 致谢
+
+主要贡献者：
+
+
+- TianHao Fu (Peking University) and [TELOS-syslab](https://github.com/TELOS-syslab/)
+- Parth Parikh (KEYSIGHT)
+- Sarah-Michelle Hammer & Ziyi Wang (TU-Berlin)
+- Xinyue Li (BUPT)
+- Tong Chen (Zhejiang University)
+- Ming Wang (BUPT)
+- Tao Jiang (Institute of Computing Technology, Chinese Academy of Sciences)
+
+以及其他个人贡献者——详见 [SimAI 贡献者列表](https://github.com/aliyun/SimAI/graphs/contributors)。
+
+特别感谢 Chenning Li（MIT CSAIL）发起将 SimAI 集成到 [M4](https://github.com/netiken/m4) 的合作。
+
+---
+
+## 参与贡献
+
+我们热烈欢迎社区贡献！详见 [贡献指南](../developer_guide/contributing.md)。
diff --git a/docs/zh/components/aicb.md b/docs/zh/components/aicb.md
new file mode 100644
index 00000000..b0842a18
--- /dev/null
+++ b/docs/zh/components/aicb.md
@@ -0,0 +1,212 @@
+# AICB — AI 通信基准测试
+
+**仓库**: [aliyun/aicb](https://github.com/aliyun/aicb) | **语言**: Python
+
+AICB 是面向 AI 场景的专用通信基准测试套件。它能生成与真实 LLM 训练和推理流程对齐的通信工作负载。
+
+---
+
+## 简介
+
+AICB（Artificial Intelligence Communication Benchmark）生成与真实应用精准对齐的通信工作负载模式，支持：
+
+- 基准测试和调优 GPU 集群通信系统
+- 研究特定模型配置的通信模式
+- 为 SimAI 等仿真器生成工作负载
+
+---
+
+## 基准测试套件
+
+AICB 提供 10 个预配置的基准测试案例，覆盖典型 LLM 配置：
+
+| 编号 | 模型 | 序列长度 | 框架 | TP | PP | SP | MoE |
+|----|-------|-----------|-----------|----|----|-----|-----|
+| 1 | LLaMA-7B | 2048 | Megatron | 1 | 1 | - | - |
+| 2 | GPT-13B | 2048 | Megatron | 2 | 1 | 是 | - |
+| 3 | GPT-22B | 2048 | Megatron | 4 | 1 | - | - |
+| 4 | LLaMA-65B | 4096 | Megatron | 8 | 2 | 是 | - |
+| 5 | GPT-175B | 2048 | Megatron | 8 | 8 | 是 | - |
+| 6 | GPT-175B | 2048 | Megatron | 8 | 8 | - | - |
+| 7 | Llama3-405B | 8192 | Megatron | 8 | 16 | 是 | - |
+| 8 | LLaMA-7B | 4096 | DeepSpeed | 1 | 1 | - | Zero-2 |
+| 9 | LLaMA-65B | 4096 | DeepSpeed | 1 | 1 | - | Zero-3 |
+| 10 | Mistral-8x7B | 2048 | Megatron | 2 | 1 | 是 | 8 experts |
+
+---
+
+## 环境搭建
+
+### Docker
+
+```bash
+docker build -t aicb:v0.0.1 .
+docker run --gpus all --net host --shm-size 16g -it --rm aicb:v0.0.1
+```
+
+### 本地环境
+
+要求：Python >= 3.8、CUDA >= 11.8、PyTorch >= 2.0.0、NVIDIA APEX
+
+### NGC 容器
+
+```bash
+docker pull nvcr.io/nvidia/pytorch:xx.xx-py3
+docker run --gpus all -it --rm -v /path/to/aicb:/workspace/aicb nvcr.io/nvidia/pytorch:xx.xx-py3
+```
+
+> **注意**：推理工作负载 Profiling 需要 NVIDIA Hopper (SM90) 或 Blackwell (SM100) GPU。
+
+---
+
+## 物理集群执行
+
+### 环境变量
+
+| 参数 | 说明 |
+|-----------|-------------|
+| `nnodes` | 节点数量 |
+| `node_rank` | 当前节点编号 |
+| `nproc_per_node` | 每节点 GPU 数 |
+| `master_addr` | 主节点地址 |
+| `master_port` | 主节点端口 |
+
+### 运行 Megatron 工作负载
+
+```bash
+sh scripts/megatron_gpt.sh \
+ --nnodes 1 --node_rank 0 --nproc_per_node 8 \
+ --master_addr localhost --master_port 29500 \
+ -m 7 --world_size 8 --tensor_model_parallel_size 2 --pipeline_model_parallel 1 \
+ --frame Megatron --global_batch 16 --micro_batch 1 \
+ --seq_length 2048 --swiglu --use_flash_attn --aiob_enable
+```
+
+### 运行 MoE 工作负载
+
+```bash
+sh scripts/megatron_gpt.sh \
+ -m moe --world_size 8 --tensor_model_parallel_size 4 \
+ --moe_enable --expert_model_parallel_size 1 \
+ --num_experts 4 --moe_router_topk 2 \
+ --frame Megatron --global_batch 16 --micro_batch 1 \
+ --sp --grouped_gemm --aiob_enable --swiglu --use_flash_attn
+```
+
+### 运行 DeepSeek 工作负载
+
+```bash
+sh scripts/megatron_gpt.sh \
+ --frame DeepSeek -m deepseek \
+ --tensor_model_parallel_size 4 --moe_enable \
+ --expert_model_parallel_size 1 --num_experts 4 \
+ --global_batch 4 --micro_batch 1 --world_size 4 \
+ --num_layers 10 --sp --swiglu --aiob_enable
+```
+
+---
+
+## 训练工作负载生成
+
+为 SimAI 仿真生成工作负载文件：
+
+```bash
+python -m workload_generator.SimAI_training_workload_generator \
+ --model_name GPT-13B --frame=Megatron \
+ --world_size=16 --tensor_model_parallel_size=2 --pipeline_model_parallel=1 \
+ --global_batch=16 --micro_batch=1 --num_layers=40 --seq_length=2048 \
+ --hidden_size=5120 --epoch_num=1 --num_attention_heads=40 \
+ --aiob_enable --use_flash_attn --swiglu
+```
+
+输出保存在 `results/mocked_workload/`。
+
+---
+
+## 推理工作负载生成
+
+AICB 为以下模型生成带 Prefill/Decode 阶段分离的推理工作负载：
+
+| 模型 | 注意力架构 | MoE 专家数 |
+|-------|-----------|-------------|
+| DeepSeek-V3-671B | MLA | 256 路由 + 1 共享 |
+| Qwen3-MoE-235B | MHA/GQA | 128 路由 |
+| Qwen3-Next-80B | 混合（全注意力 + 线性注意力） | 512 路由 |
+
+需要硬件加速库：[DeepGEMM](https://github.com/deepseek-ai/DeepGEMM)、[FlashMLA](https://github.com/deepseek-ai/FlashMLA)、[FlashInfer](https://github.com/flashinfer-ai/flashinfer)。
+
+---
+
+## AIOB：计算性能分析
+
+AIOB 可采集实际 GPU 计算耗时并嵌入工作负载：
+
+- `--aiob_enable` — 在当前 GPU 上进行 Profiling
+- `--comp_filepath
+
++ English ｜ English +
+ +[](../../LICENSE) +[](https://ennanzhai.github.io/pub/nsdi25spring-simai.pdf) + +**SimAI** 是阿里云开源的业界首个全栈高精度 AI 大规模**推理**与**训练**仿真器。它提供了对 LLM 训练和推理全流程的详细建模与仿真，涵盖框架层、集合通信层和网络传输层，提供端到端的性能数据。 + +SimAI 使研究人员能够： + +- 分析推理/训练过程细节 +- 评估特定条件下 AI 任务的时间消耗 +- 评估各种算法优化带来的端到端性能提升（框架参数、集合通信算法、网络协议、拥塞控制、路由、拓扑等） + +--- + +## 文档概览 + +| 章节 | 说明 | +|------|------| +| [快速入门](getting_started/index.md) | 安装、环境搭建、快速开始 | +| [用户指南](user_guide/index.md) | SimAI-Analytical、SimAI-Simulation、SimAI-Physical、推理仿真的详细使用方法 | +| [组件详情](components/index.md) | 各子模块详细文档：AICB、SimCCL、astra-sim、ns-3、vidur | +| [技术参考](technical_reference/index.md) | GPU 显存模块、CLI 参数、配置文件参考 | +| [基准测试](benchmarking/index.md) | 四场景端到端测试套件及基准测试结果 | +| [开发者指南](developer_guide/index.md) | 架构、贡献指南、添加模型、扩展 NS-3 | +| [社区](community/index.md) | 活动、联系方式、引用 | + +--- + +## 系统架构 + +``` + |--- AICB （工作负载生成 & 计算性能分析） +SimAI --|--- SimCCL （集合通信算法分析） + |--- astra-sim-alibabacloud （仿真引擎：Analytical / Simulation / Physical） + |--- ns-3-alibabacloud （NS-3 网络后端） + |--- vidur-alibabacloud （多请求推理调度 & 显存管理） +``` + + + +--- + +## 三种运行模式 + +| 模式 | 说明 | 适用场景 | +|------|------|----------| +| **SimAI-Analytical** | 使用总线带宽（busbw）估算集合通信时间的快速仿真 | 性能分析、并行参数优化、scale-up 探索 | +| **SimAI-Simulation** | 使用 NS-3 网络后端的全栈仿真，提供细粒度网络建模 | CC 算法研究、网络协议评估、新架构设计 | +| **SimAI-Physical** *(Beta)* | 在 CPU RDMA 集群上生成物理流量 | NIC 行为研究 | + +--- + +## 支持的模型 + +- **DeepSeek-V3-671B** — MLA 注意力，256 个路由专家 +- **Qwen3-MoE-235B** — MHA/GQA，128 个路由专家 +- **Qwen3-Next-80B** — 混合全注意力 + 线性注意力，512 个路由专家 +- **Meta-Llama-3-8B / 70B**、**Llama-2-7b / 70b**、**CodeLlama-34b**、**InternLM-20B**、**Qwen-72B** + +--- + +## 快速链接 + +- [GitHub 仓库](https://github.com/aliyun/SimAI) +- [NSDI'25 论文 (PDF)](https://ennanzhai.github.io/pub/nsdi25spring-simai.pdf) +- [演示文稿](../../docs/SimAI_Intro_Online.pdf) +- [技术报告 (1.6)](../SimAI_1.6_Tech_Report.md) +- [贡献指南](../../CONTRIBUTING.md) diff --git a/docs/zh/technical_reference/cli_reference.md b/docs/zh/technical_reference/cli_reference.md new file mode 100644 index 00000000..3c1e1ad2 --- /dev/null +++ b/docs/zh/technical_reference/cli_reference.md @@ -0,0 +1,185 @@ +# CLI 参考 + +SimAI 所有工具的完整命令行参数参考。 + +--- + +## SimAI-Analytical + +**二进制**: `bin/SimAI_analytical` + +### 必需参数 + +| 标志 | 长格式 | 说明 | +|------|-----------|-------------| +| `-w` | `--workload` | 工作负载文件路径 | +| `-g` | `--gpus` | 仿真 GPU 规模 | +| `-g_p_s` | `--gpus-per-server` | Scale-up 大小（每服务器 GPU 数） | +| `-r` | `--result` | 输出文件路径和前缀（默认：`./results/`） | +| `-busbw` | `--bus-bandwidth` | busbw.yaml 文件路径 | + +### 可选参数 + +| 标志 | 长格式 | 说明 | +|------|-----------|-------------| +| `-v` | `--visual` | 生成可视化文件 | +| `-dp_o` | `--dp-overlap-ratio` | DP 重叠比例 [0.0-1.0] | +| `-ep_o` | `--ep-overlap-ratio` | EP 重叠比例 [0.0-1.0] | +| `-tp_o` | `--tp-overlap-ratio` | TP 重叠比例 [0.0-1.0] | +| `-pp_o` | `--pp-overlap-ratio` | PP 重叠比例 [0.0-1.0] | + +### 自动 Busbw 计算 + +| 标志 | 说明 | +|------|-------------| +| `-nv` | NVLink 带宽（GB/s） | +| `-nic` | NIC 带宽（GB/s） | +| `-n_p_s` | 每服务器 NIC 数 | + +--- + +## SimAI-Simulation + +**二进制**: `bin/SimAI_simulator` + +### 环境变量 + +| 变量 | 说明 | 默认值 | +|----------|-------------|---------| +| `AS_LOG_LEVEL` | 日志级别：DEBUG/INFO/WARNING/ERROR | `INFO` | +| `AS_PXN_ENABLE` | 启用 PXN | `0` | +| `AS_NVLS_ENABLE` | 启用 NVLS | `0` | +| `AS_SEND_LAT` | 发送延迟（us） | `6` | +| `AS_NVLSTREE_ENABLE` | 启用 NVLS Tree | `false` | + +### 参数 + +| 标志 | 长格式 | 说明 | 默认值 | +|------|-----------|-------------|---------| +| `-t` | `--thread` | 线程数 | `1` | +| `-w` | `--workload` | 工作负载路径 | 必需 | +| `-n` | `--network-topo` | 拓扑文件路径 | 必需 | +| `-c` | `--config` | SimAI.conf 路径 | 必需 | + +--- + +## SimAI-Physical + +**二进制**: `bin/SimAI_phynet` + +| 参数 | 说明 | 默认值 | +|-----------|-------------|---------| +| `hostlist` | 主机 IP 列表路径 | 必需 | +| `-w` / `--workload` | 工作负载文件路径 | `./microAllReduce.txt` | +| `-i` / `--gid_index` | RDMA 的 GID 索引 | `0` | +| `-g` / `--gpus` | GPU 数量 | `8` | + +--- + +## 拓扑生成器 + +**脚本**: `astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py` + +| 层级 | 标志 | 说明 | +|-------|------|-------------| +| 全局 | `-topo` | 模板：Spectrum-X / AlibabaHPN / DCN+ | +| | `-g` | GPU 数量 | +| | `--dp` | 启用双 Plane | +| | `--ro` | 启用 Rail-optimized | +| | `--dt` | 启用双 ToR | +| | `-er` | 错误率 | +| 服务器内 | `-gps` | 每服务器 GPU 数 | +| | `-gt` | GPU 型号（A100/H100） | +| | `-nsps` | 每服务器 NV Switch 数 | +| | `-nvbw` | NVLink 带宽 | +| | `-nl` | NVLink 延迟 | +| | `-l` | NIC 延迟 | +| Segment 内 | `-bw` | NIC 到 ASW 带宽 | +| | `-asw` | ASW 交换机数量 | +| | `-nps` | 每交换机 NIC 数 | +| Pod 内 | `-psn` | PSW 交换机数量 | +| | `-apbw` | ASW 到 PSW 带宽 | +| | `-app` | 每 PSW 的 ASW 数 | + +--- + +## AICB 工作负载生成器 + +**脚本**: `scripts/megatron_workload_with_aiob.sh` 或 `python -m workload_generator.SimAI_training_workload_generator` + +### 核心参数 + +| 参数 | 说明 | +|-----------|-------------| +| `--frame` | 框架：Megatron / DeepSpeed / DeepSeek | +| `-m` / `--model_size` | 模型大小：7/13/22/175/moe/deepseek | +| `--world_size` | 总 GPU 数量 | +| `--global_batch` | 总批量大小 | +| `--micro_batch` | 微批量大小 | +| `--seq_length` | 序列长度 | +| `--epoch_num` | 迭代次数 | + +### 并行参数 + +| 参数 | 说明 | +|-----------|-------------| +| `--tensor_model_parallel_size` | TP 度 | +| `--pipeline_model_parallel` | PP 度 | +| `--expert_model_parallel_size` | EP 度 | +| `--enable_sequence_parallel` | 启用 SP | + +### 模型参数 + +| 参数 | 说明 | +|-----------|-------------| +| `--num_layers` | Transformer 层数 | +| `--hidden_size` | 隐藏层大小 | +| `--num_attention_heads` | 注意力头数 | +| `--ffn_hidden_size` | FFN 隐藏层大小 | +| `--vocab_size` | 词表大小 | + +### MoE 参数 + +| 参数 | 说明 | +|-----------|-------------| +| `--moe_enable` | 启用 MoE | +| `--num_experts` | 专家数量 | +| `--moe_router_topk` | 每 Token 专家数 | +| `--moe_grouped_gemm` | 启用分组 GEMM | + +### DeepSeek 参数 + +| 参数 | 说明 | +|-----------|-------------| +| `--qk_rope_dim` | QK 的 RoPE 维度 | +| `--qk_nope_dim` | QK 的非 RoPE 维度 | +| `--q_lora_rank` | Q LoRA 秩 | +| `--kv_lora_rank` | KV LoRA 秩 | +| `--v_head_dim` | V Head 维度 | +| `--n_shared_expert` | 每 MoE 层共享专家数 | +| `--n_dense_layer` | 稠密层数 | + +### 优化参数 + +| 参数 | 说明 | +|-----------|-------------| +| `--use_flash_attn` | FlashAttention | +| `--swiglu` | SwiGLU 激活函数 | +| `--aiob_enable` | AIOB 计算 Profiling | +| `--comp_filepath` | 预计算时间文件 | + +--- + +## Vidur 推理仿真 + +**命令**: `python -m vidur.main` + +运行 `python -m vidur.main -h` 查看完整参数列表。关键参数见 [vidur 组件页面](../components/vidur.md)。 + +--- + +## 相关文档 + +- [配置文件参考](configuration.md) — 配置文件格式 +- [SimAI-Analytical 指南](../user_guide/simai_analytical.md) — 使用示例 +- [AICB 组件](../components/aicb.md) — 完整参数详情 diff --git a/docs/zh/technical_reference/configuration.md b/docs/zh/technical_reference/configuration.md new file mode 100644 index 00000000..6486f738 --- /dev/null +++ b/docs/zh/technical_reference/configuration.md @@ -0,0 +1,163 @@ +# 配置文件参考 + +本文档涵盖 SimAI 使用的所有配置文件。 + +--- + +## SimAI.conf + +**路径**: `astra-sim-alibabacloud/inputs/config/SimAI.conf` + +SimAI-Analytical 和 SimAI-Simulation 模式共用的主仿真配置文件，控制通信算法、缓冲区大小和时序参数。 + +--- + +## busbw.yaml + +**路径**: `example/busbw.yaml` + +SimAI-Analytical 使用，用于指定不同通信组和集合操作的总线带宽。 + +### 格式 + +```yaml +test +TP: + allreduce,: 300 # TP 组 AllReduce busbw 300GB/s + allgather,: 280 + reducescatter,: 280 + alltoall,: 230 +DP: + allreduce,: null # null = 该组不使用此操作 + allgather,: 380 + reducescatter,: 380 + alltoall,: null +EP: + allreduce,: null + allgather,: 45 + reducescatter,: 45 + alltoall,: 80 +``` + +### 通信组 + +| 组 | 说明 | +|-------|-------------| +| `TP` | 张量并行 — 服务器内 NVLink 通信 | +| `DP` | 数据并行 — 服务器间 RDMA 通信 | +| `EP` | 专家并行 — MoE 专家通信 | + +### 集合操作 + +| 操作 | 说明 | +|-----------|-------------| +| `allreduce` | 归约 + 广播到所有 Rank | +| `allgather` | 从所有 Rank 收集数据 | +| `reducescatter` | 归约并分发 | +| `alltoall` | 全对全个性化交换 | + +对特定组中不使用的操作，将值设为 `null`。 + +--- + +## 拓扑文件 + +由 `gen_Topo_Template.py` 生成，拓扑文件为 SimAI-Simulation 定义网络结构。 + +### 生成 + +```bash +python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py \ + -topo Spectrum-X -g 128 -gt A100 -bw 100Gbps -nvbw 2400Gbps +``` + +输出文件以参数命名，如 `Spectrum-X_128g_8gps_100Gbps_A100`。 + +### 模板默认值 + +| 模板 | GPU 数 | 拓扑 | 带宽 | GPU 型号 | +|----------|------|------|-----------|----------| +| Spectrum-X | 4096 | Rail-optimized，单 ToR | 400Gbps | H100 | +| AlibabaHPN（单 Plane） | 15360 | Rail-optimized，双 ToR | 200Gbps | H100 | +| AlibabaHPN（双 Plane） | 15360 | Rail-optimized，双 ToR，双 Plane | 200Gbps | H100 | +| DCN+（单 ToR） | 512 | 非 Rail-optimized | 400Gbps | A100 | +| DCN+（双 ToR） | 512 | 非 Rail-optimized，双 ToR | 200Gbps | H100 | + +--- + +## 模型配置文件 + +### 推理模型配置 + +位于 `vidur-alibabacloud/data/hf_configs/`： + +| 模型 | 配置文件 | +|-------|------------| +| DeepSeek-V3-671B | `deepseek_v3_config.json` | +| Qwen3-MoE-235B | `qwen3_moe_config.json` | +| Qwen3-Next-80B | `qwen3-next-80B-A3B_config.json` | + +这些文件遵循 HuggingFace `config.json` 格式，定义模型架构参数。 + +### Profiling 数据 + +位于 `vidur-alibabacloud/data/profiling/`： + +``` +profiling/ +├── compute/ +│ ├── a100/ +│ │ └──+ 中文 ｜ English +
+# Vidur-AlibabaCloud + +[](https://www.python.org/downloads/) +[](LICENSE) + +Vidur ([original](https://github.com/microsoft/vidur)) is a simulation framework for large language model (LLM) inference systems. +**Vidur-AlibabaCloud** (this repository) is a customized version optimized for Alibaba Cloud **SimAI** scenarios. It supports advanced features such as **Prefill–Decode (PD) disaggregation** and includes dedicated adaptations for SOTA LLM models including **DeepSeek-V3-671B**, **Qwen3-MoE-235B**, **Qwen3-Next-80B**, and others. -Vidur ([original](https://github.com/microsoft/vidur)) is a simulation framework for large language model (LLM) inference systems. -**Vidur-AlibabaCloud** (this repository) is a customized version optimized for Alibaba Cloud **SimAI** scenarios. It supports advanced features such as **Prefill–Decode (PD) disaggregation** and includes dedicated adaptations for state-of-the-art (SOTA) LLM models including **DeepSeek-V3-671B**, **Qwen3-MoE-235B**, **Qwen3-Next-80B**, and other models. + +--- + +## Table of Contents + +- [Key Features](#key-features) +- [GPU Memory Calculation](#gpu-memory-calculation) +- [Supported Models](#supported-models) +- [Environment Setup](#-environment-setup) +- [Running Examples](#%EF%B8%8F-running-examples) + - [4-Scenario Configuration](#4-scenario-configuration) + - [Output Files](#output-files) +- [Key Input Parameters](#-key-input-parameter-reference) +- [Key Output Interpretation](#-key-output-interpretation) +- [Known Issues](#%EF%B8%8F-known-issues) +- [Help](#-help) --- ## Key Features -+ **Prefill–Decode (PD) Separation** – Enables running the prefill and decode stages on different nodes, allowing elastic resource allocation and performance isolation. -(Inspired by [splitwise-sim](https://github.com/Mutinifni/splitwise-sim)). -+ **Flexible Parallelism** – Supports: - - **Data Parallel (DP)** - - **Tensor Parallel (TP)** - - **Pipeline Parallel (PP)** - - **Expert Parallel (EP)** (support in progress) -Works for both **dense** and **Mixture-of-Experts (MoE)** models (MoE support in progress). -+ **Multiple Execution-Time Prediction Backends** – Choose from: - - **AICB/AIOB** - Partially supports computation kernels and TP, DP, PP, EP communication size for DeepSeek-V3-671B, Qwen3-Moe-235B, Qwen3-Next-80B - - **SimAi_simulation** – SimAI NS-3-based network simulation (supports TP) - - **SimAi_analytical** – SimAI analytical performance model (supports TP) - - **Native Vidur [original]** – Supports TP, DP, PP -+ **Workload Generation & Replay** – Replay real-world traces or generate synthetic requests using fixed or Poisson distributions. -+ **Fine-Grained Metrics** – Records: - - TTFT – Time to First Token - - TBT / TPOT – Time Between Tokens / Time Per Output Token - - End-to-end latency - - Communication cost - - Computation cost - - Scheduling delay + +- **Prefill–Decode (PD) Separation** — Enables running the prefill and decode stages on different nodes, allowing elastic resource allocation and performance isolation. + (Inspired by [splitwise-sim](https://github.com/Mutinifni/splitwise-sim)) +- **Flexible Parallelism** — Supports: + - **Data Parallel (DP)** + - **Tensor Parallel (TP)** + - **Pipeline Parallel (PP)** + - **Expert Parallel (EP)** (support in progress) + + Works for both **dense** and **Mixture-of-Experts (MoE)** models (MoE support in progress). +- **Multiple Execution-Time Prediction Backends** — Choose from: + - **AICB/AIOB** — Partially supports computation kernels and TP, DP, PP, EP communication size for DeepSeek-V3-671B, Qwen3-MoE-235B, Qwen3-Next-80B + - **SimAI Simulation** — SimAI NS-3-based network simulation (supports TP) + - **SimAI Analytical** — SimAI analytical performance model (supports TP) + - **Native Vidur [original]** — Supports TP, DP, PP +- **Workload Generation & Replay** — Replay real-world traces or generate synthetic requests using fixed or Poisson distributions. +- **Fine-Grained Metrics** — Records: + - TTFT — Time to First Token + - TBT / TPOT — Time Between Tokens / Time Per Output Token + - End-to-end latency + - Communication cost + - Computation cost + - Scheduling delay + +--- + +## GPU Memory Calculation + +This module provides accurate GPU memory estimation for modern MoE (Mixture-of-Experts) models during inference simulation, covering **model parameter memory**, **KV cache memory**, and **maximum batch size** calculation under Prefill–Decode (PD) disaggregation. + +### Supported Attention Architectures + +| Architecture | Model | Description | +|---|---|---| +| **MLA** (Multi-head Latent Attention) | DeepSeek-V3-671B | Uses LoRA-compressed KV cache (`kv_lora_rank` + `qk_rope_head_dim`) for reduced memory footprint | +| **MHA / GQA** (Multi-Head / Grouped-Query Attention) | Qwen3-MoE-235B | Standard KV cache with `num_kv_heads * head_dim` per token per layer | +| **Hybrid Full + Linear Attention** | Qwen3-Next-80B | Alternates between full attention and linear (GDN) attention every 4 layers | + +### Key Components + +- **`ParamCounter`** (`vidur/utils/param_counter.py`) — Computes per-layer and per-device parameter counts for MLA, MHA/GQA, linear attention, and MoE expert weights, with FP8 quantization support. Under PD disaggregation, it returns separate `(total_params, prefill_params, decode_params)` based on `prefill_world_size` / `decode_world_size`. +- **`MemoryPlanner`** (`vidur/scheduler/utils/memory_planner.py`) — Plans GPU memory budget: `available = GPU_mem * (1 - margin) - param_mem`, then computes KV cache capacity and maximum concurrent requests. Includes OOM detection with actionable suggestions. +- **Per-request KV cache tracking** (`vidur/entities/replica.py`) — Allocates and releases KV cache memory on a per-request basis, enabling accurate remaining-capacity queries at runtime. + +### References & Acknowledgments + +The GPU memory calculation module was developed with reference to the following works: + +- [InferSim](https://github.com/alibaba/InferSim) — Parameter counting and KV cache estimation methodology +- [DeepSeek V3 Parameter Size Analysis](https://yangwenbo.com/articles/deepseek-v3-parameter-size.html) — DeepSeek V3 MLA parameter derivation +- [DeepSeek V3 Parameter Derivation (Chinese)](https://zhuanlan.zhihu.com/p/21455638257) — Detailed MLA weight decomposition + +We gratefully acknowledge these resources for providing the foundational analysis that guided our implementation. --- ## Supported Models -+ **DeepSeek-V3-671B** (SimAI PP/EP communication、GPU memory allocation module adaptations in progress) -+ **Qwen3-Moe-235B**, **Qwen3-Next-80B** (SimAI PP/EP communication、GPU memory allocation module adaptations in progress) -+ **meta-llama/Meta-Llama-3-8B** / **Meta-Llama-3-70B** -+ **meta-llama/Llama-2-7b-hf** / **Llama-2-70b-hf** -+ **codellama/CodeLlama-34b-Instruct-hf** -+ **internlm/internlm-20b** -+ **Qwen/Qwen-72B** + +- **DeepSeek-V3-671B** (SimAI PP/EP communication and GPU memory allocation module adaptations in progress) +- **Qwen3-MoE-235B**, **Qwen3-Next-80B** (SimAI PP/EP communication and GPU memory allocation module adaptations in progress) +- **meta-llama/Meta-Llama-3-8B** / **Meta-Llama-3-70B** +- **meta-llama/Llama-2-7b-hf** / **Llama-2-70b-hf** +- **codellama/CodeLlama-34b-Instruct-hf** +- **internlm/internlm-20b** +- **Qwen/Qwen-72B** --- ## 📦 Environment Setup + ### 1. Create Conda Environment + ```bash conda env create -p ./env -f ./environment.yml ``` ### 2. (Optional) Update Dev Dependencies + ```bash conda env update -f environment-dev.yml ``` ### 3. Activate Environment + ```bash conda activate vidur ``` ### 4. Install Python Dependencies (Using Alibaba Cloud PyPI Mirror) + ```bash pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ pip install -r requirements-dev.txt -i https://mirrors.aliyun.com/pypi/simple/ @@ -66,13 +127,16 @@ pip install -r requirements-dev.txt -i https://mirrors.aliyun.com/pypi/simple/ --- -## ▶️ Running Example -### Run DeepSeek-671B **with** AICB -**Requirements: **SimAI and AICB Docker environment (see [README](../README.md) for setup instructions). +## ▶️ Running Examples + +### Run DeepSeek-671B with AICB -After setting up the environment, run the following commands: +**Requirements:** SimAI and AICB Docker environment (see [README](../README.md) for setup instructions). + +After setting up the environment, run the following commands: + +#### DeepSeek-671B with AICB (Fixed Length Generator) -#### Run DeepSeek-671B **with** AICB (Fixed Length Generator) ```bash cd SimAI/vidur-alibabacloud @@ -94,10 +158,11 @@ python -m vidur.main --replica_config_pd_p2p_comm_bandwidth 800 \ --replica_config_tensor_parallel_size 2 \ --replica_config_num_pipeline_stages 1 \ --replica_config_expert_model_parallel_size 8 \ - --random_forrest_execution_time_predictor_config_backend aicb + --random_forrest_execution_time_predictor_config_backend aicb ``` -#### Run DeepSeek-671B **with** AICB (Trace Length Generator) +#### DeepSeek-671B with AICB (Trace Length Generator) + ```bash cd SimAI/vidur-alibabacloud @@ -124,11 +189,9 @@ python -m vidur.main \ ``` > ✅ Full parameter descriptions are available via `python -m vidur.main -h`. -> +### Run Llama-3-8B with SimAI Simulation - -### Run Llama-3-8B **with** simai_simulation ```bash cd SimAI @@ -136,8 +199,8 @@ cd SimAI ./scripts/build.sh -c ns3 # Create network topo (Spectrum-X_128g_8gps_100Gbps_A100) -python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py -topo Spectrum-X -g 128 -gt A100 -bw 100Gbps -nvbw 2400Gbps - +python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py \ + -topo Spectrum-X -g 128 -gt A100 -bw 100Gbps -nvbw 2400Gbps cd SimAI/vidur-alibabacloud @@ -163,18 +226,16 @@ python -m vidur.main \ --random_forrest_execution_time_predictor_config_backend simai_simulation \ --random_forrest_execution_time_predictor_config_simai_dir ../ \ --random_forrest_execution_time_predictor_config_simai_simulation_topo ../Spectrum-X_128g_8gps_100Gbps_A100 \ - --random_forrest_execution_time_predictor_config_simai_simulation_config ../astra-sim-alibabacloud/inputs/config/SimAI.conf + --random_forrest_execution_time_predictor_config_simai_simulation_config ../astra-sim-alibabacloud/inputs/config/SimAI.conf ``` -> -> +### Run Llama-3-8B with SimAI Analytical -### Run Llama-3-8B **with** simai_analytical ```bash cd SimAI # Compile SimAI-Analytical -$ ./scripts/build.sh -c analytical +./scripts/build.sh -c analytical cd SimAI/vidur-alibabacloud @@ -200,10 +261,8 @@ python -m vidur.main \ --random_forrest_execution_time_predictor_config_backend simai_analytical ``` -> -> +### Run Llama-3-8B with Native Vidur [original] -### Run Llama-3-8B **with** native Vidur [original] ```bash cd SimAI/vidur-alibabacloud @@ -229,125 +288,172 @@ python -m vidur.main \ --random_forrest_execution_time_predictor_config_backend vidur ``` -> -> +### Run 4-Scenario Suite +For a quick validation of all supported configurations, use the bundled test script: +```bash +bash examples/vidur-ali-scenarios/run_scenarios.sh --all +``` + +See `bash examples/vidur-ali-scenarios/run_scenarios.sh --help` for details. + +#### 4-Scenario Configuration + +The following scenarios are pre-configured in `run_scenarios.sh`. All scenarios share the hardware configuration below. + +**Shared Hardware Configuration:** +- GPU: H20 (h20_dgx), NVLink: 1600 Gbps, RDMA: 800 Gbps +- PD P2P bandwidth: 800 Gbps, dtype: fp8 +- Request: Poisson QPS=100, 4 requests, fixed prefill=100 / decode=8 tokens + +| Scenario | Model | PD Separation | World Size | TP | PP | EP | Global Scheduler | +|----------|-------|---------------|------------|----|----|------------|------------------| +| 1 | Qwen3-Next-80B (MoE) | No | 32 (dp=32) | 1 | 1 | 1 (default) | lor | +| 2 | Qwen3-Next-80B (MoE) | Yes (P=2, D=6) | 8 | 1 | 1 | 1 (default) | split_wise | +| 3 | DeepSeek-671B (MoE) | Yes (P=2, D=6) | 8 | 8 | 1 | 8 | split_wise | +| 4 | Qwen3-MoE-235B (MoE) | Yes (P=2, D=6) | 8 | 4 | 1 | 4 | split_wise | + +> **Note:** All four models use Mixture-of-Experts (MoE) architecture. The EP column reflects the explicit `--replica_config_expert_model_parallel_size` value set in the script; scenarios without an explicit EP setting use the default value of 1. + +#### Output Files + +**Output path depends on how you run the simulation:** + +- **`run_scenarios.sh`** --- outputs to `examples/vidur-ali-scenarios/simulator_output/` +- **Direct `python -m vidur.main`** --- outputs to `./simulator_output/` (or the path specified by `--metrics_config_output_dir`) + +Each run produces the following directory: + +``` +⚠️ **Note**: Vidur GPU Memory management module is still under adaptation for DeepSeek-671B, Qwen3-Moe-235B, Qwen3-Next-80B | +| `--replica_config_model_name` | meta-llama/Llama-2-7b-hf | Model name (DeepSeek-671B, Qwen3-MoE-235B, Qwen3-Next-80B, etc.) | | `--replica_config_tensor_parallel_size` | 1 | Tensor parallelism size (TP) | | `--replica_config_num_pipeline_stages` | 1 | Number of pipeline stages (PP) | | `--replica_config_expert_model_parallel_size` | 1 | Expert model parallelism size (EP) | -| `--random_forrest_execution_time_predictor_config_backend` | vidur | Backend for execution time prediction
('vidur', 'simai_simulation', 'simai_analytical','aicb', etc.)
⚠️ **Note**: `simai_simulation` and `simai_analytical` currently only model TP communication and do not support pipeline or expert parallelism | -| `--random_forrest_execution_time_predictor_config_simai_dir` | `'../'` | Root directory of the SimAI simulator（default: `../`)
（only effective when `--random_forrest_execution_time_predictor_config_backend simai_simulation`） | -| `--random_forrest_execution_time_predictor_config_simai_simulation_topo` | `'../example/topo'` | Path to SimAI topology file (e.g., `'../Spectrum-X_128g_8gps_100Gbps_A100'`)（only effective when `--random_forrest_execution_time_predictor_config_backend simai_simulation`） | -| `--random_forrest_execution_time_predictor_config_simai_simulation_config` | `'../astra-sim-alibabacloud/inputs/config/SimAI.conf'` | Path to SimAI configuration file (e.g., `'../astra-sim-alibabacloud/inputs/config/SimAI.conf'`)
（only effective when `--random_forrest_execution_time_predictor_config_backend simai_simulation`） | - +| `--random_forrest_execution_time_predictor_config_backend` | vidur | Backend for execution time prediction (`vidur`, `simai_simulation`, `simai_analytical`, `aicb`, etc.). **Note:** `simai_simulation` and `simai_analytical` currently only model TP communication and do not support pipeline or expert parallelism. | +| `--random_forrest_execution_time_predictor_config_simai_dir` | `../` | Root directory of the SimAI simulator (only effective when backend = `simai_simulation`) | +| `--random_forrest_execution_time_predictor_config_simai_simulation_topo` | `../example/topo` | Path to SimAI topology file (only effective when backend = `simai_simulation`) | +| `--random_forrest_execution_time_predictor_config_simai_simulation_config` | `../astra-sim-alibabacloud/inputs/config/SimAI.conf` | Path to SimAI configuration file (only effective when backend = `simai_simulation`) | --- ## 📊 Key Output Interpretation + Simulation results are saved to: -```plain +``` ./simulator_output/YYYY-MM-DD_HH-MM-SS-XXXXXX/request_metrics.csv ``` ### Key Columns in `request_metrics.csv` + | Column | Meaning | -| --- | --- | +|--------|---------| | `arrived_at` / `prefill_arrived_at` | Timestamp when the request entered the system (in seconds). | -| `scheduled_at` | Timestamp when the request was first scheduled by the scheduler and began execution (in seconds). | +| `scheduled_at` | Timestamp when the request was first scheduled and began execution (in seconds). | | `prefill_completed_at` | Timestamp when the Prefill phase completed and the first output token was generated. | -| `decode_arrived_at` | Timestamp when the Decode phase started. In non-PD-Disaggregated setup, this typically equals `prefill_completed_at`. In PD-Disaggregated setup, it is `prefill_completed_at + pd_p2p_comm_time`. | -| `decode_time` | Duration of the Decode phase (in seconds), computed as `completed_at - decode_arrived_at` (equivalently: `request_e2e_time - prefill_e2e_time`). | -| `prefill_replica_id` | Replica ID that executed the Prefill phase (in PD-Disaggregated setup). | -| `decode_replica_id` | Replica ID that executed the Decode phase (in PD-Disaggregated setup). | +| `decode_arrived_at` | Timestamp when the Decode phase started. In non-PD-disaggregated setup, this typically equals `prefill_completed_at`. In PD-disaggregated setup, it is `prefill_completed_at + pd_p2p_comm_time`. | +| `decode_time` | Duration of the Decode phase (in seconds), computed as `completed_at - decode_arrived_at`. | +| `prefill_replica_id` | Replica ID that executed the Prefill phase (in PD-disaggregated setup). | +| `decode_replica_id` | Replica ID that executed the Decode phase (in PD-disaggregated setup). | | `request_num_prefill_tokens` | Number of input tokens (i.e., prompt length). | | `request_num_decode_tokens` | Number of output tokens (i.e., generation length). | -| `pd_p2p_comm_size` | Point-to-point communication size (in bytes) of data transferred from the Prefill node to the Decode node (KV cache, etc.) in PD-Disaggregated setup. | -| `pd_p2p_comm_time` | Point-to-point communication time (in seconds) between Prefill and Decode nodes in PD-Disaggregated setup. | +| `pd_p2p_comm_size` | P2P communication size (in bytes) of data transferred from the Prefill node to the Decode node (KV cache, etc.) in PD-disaggregated setup. | +| `pd_p2p_comm_time` | P2P communication time (in seconds) between Prefill and Decode nodes in PD-disaggregated setup. | | `completed_at` | Timestamp when the request finished processing. | | `request_execution_time` | Total actual execution time (in seconds), excluding delays due to preemption or pipeline bubbles. | | `request_preemption_time` | Time (in seconds) spent waiting due to scheduler preemption, pipeline bubbles, or other non-execution gaps. | | `request_scheduling_delay` | Scheduling delay before execution: `scheduled_at - arrived_at` (in seconds). | | `request_e2e_time` | End-to-end latency: `completed_at - arrived_at` (in seconds). | | `prefill_e2e_time` | Time To First Token (TTFT): `prefill_completed_at - arrived_at` (in seconds). | -| `tbt` | Time Between Tokens (TBT), also known as Time Per Output Token (TPOT). Computed as: `decode_time / request_num_decode_tokens` or equivalently: `(request_e2e_time - prefill_e2e_time) / request_num_decode_tokens` (in seconds/token). | +| `tbt` | Time Between Tokens (TBT), also known as TPOT. Computed as: `decode_time / request_num_decode_tokens` (in seconds/token). | +**Notes:** - **Notes**: +- All time-related fields are in **seconds (s)**, based on monotonic clock or Unix timestamps. +- In non-PD-separated deployments, `prefill_replica_id` and `decode_replica_id` are typically identical. +- If `request_num_decode_tokens = 0`, `tbt` is undefined (may be recorded as `NaN` or `0`). +- TBT is not yet logged in `request_metrics.csv`; it can be computed manually for now. -+ All time-related fields are in **seconds (s)**, based on monotonic clock or Unix timestamps. -+ In non-PD-separated deployments, `prefill_replica_id` and `decode_replica_id` are typically identical. -+ If `request_num_decode_tokens = 0`, `tbt` is undefined (may be recorded as `NaN` or `0`). -+ **TBT is not yet logged in request_metrics.csv; it can be computed manually for now.** +### Sample Row (`request_metrics.csv`) -### Sample Row (request_metrics.csv) -```plain +``` Request Id,request_e2e_time,...,arrived_at,prefill_arrived_at,scheduled_at,prefill_completed_at,decode_arrived_at,completed_at,...,prefill_replica_id,decode_replica_id,pd_p2p_comm_size,pd_p2p_comm_time,... 0,0.03607,...,0.0102006,0.0102006,0.0102006,0.0102265,0.0433997,0.0462744,...,0,2,3561947136,0.0331732,... ``` --- -## ⚠️ Known Issue: Plotting Warning +## ⚠️ Known Issues + +### Plotting Warning + You may see this error at exit: -```plain +``` RuntimeError: Kaleido requires Google Chrome to be installed. ``` -This occurs because the simulator tries to generate PNG plots but lacks Chrome. -✅ **Important**: This **does NOT affect** the generation of `request_metrics.csv`. +This occurs because the simulator tries to generate PNG plots but lacks Chrome. +**Important:** This does **NOT** affect the generation of `request_metrics.csv`. -### Solutions: -1. **Ignore it** – CSV output is unaffected. -2. **Install Chrome**: - -```bash -plotly_get_chrome -``` +**Solutions:** +1. **Ignore it** — CSV output is unaffected. +2. **Install Chrome:** + ```bash + plotly_get_chrome + ``` 3. **Disable plotting** (not recommended): Comment out these lines in `vidur/simulator.py`: - -```python -# self._metric_store.plot() -# logger.info("Metrics written") -``` - -> ⚠️ Disabling plotting will skip all visual outputs and request_metrics.csv. -> + ```python + # self._metric_store.plot() + # logger.info("Metrics written") + ``` + > Disabling plotting will skip all visual outputs and `request_metrics.csv`. --- ## 📚 Help + View all CLI options: ```bash python -m vidur.main -h ``` - ---- - diff --git a/vidur-alibabacloud/README_CN.md b/vidur-alibabacloud/README_CN.md new file mode 100644 index 00000000..372f8303 --- /dev/null +++ b/vidur-alibabacloud/README_CN.md @@ -0,0 +1,458 @@ +
+ 中文 ｜ English +
+ +# Vidur-AlibabaCloud + +[](https://www.python.org/downloads/) +[](LICENSE) + +Vidur（[原版](https://github.com/microsoft/vidur)）是一个大语言模型（LLM）推理系统的模拟框架。 +**Vidur-AlibabaCloud**（本仓库）是针对阿里云 **SimAI** 场景优化的定制版本。支持 **Prefill–Decode（PD）分离**等高级特性，并针对 **DeepSeek-V3-671B**、**Qwen3-MoE-235B**、**Qwen3-Next-80B** 等 SOTA 大模型进行了专门适配。 + +--- + +## 目录 + +- [主要特性](#主要特性) +- [GPU 显存计算模块](#gpu-显存计算模块) +- [支持的模型](#支持的模型) +- [📦 环境配置](#-环境配置) +- [▶️ 运行示例](#️-运行示例) + - [四场景配置说明](#四场景配置说明) + - [输出文件说明](#输出文件说明) +- [🔧 关键输入参数参考](#-关键输入参数参考) +- [📊 输出结果解读](#-输出结果解读) +- [⚠️ 已知问题](#️-已知问题) +- [📚 帮助](#-帮助) + +--- + +## 主要特性 + +- **Prefill–Decode（PD）分离** — 支持 prefill 和 decode 阶段在不同节点运行，实现弹性资源分配和性能隔离。 + （参考 [splitwise-sim](https://github.com/Mutinifni/splitwise-sim)） +- **灵活的并行策略** — 支持： + - **数据并行（DP）** + - **张量并行（TP）** + - **流水线并行（PP）** + - **专家并行（EP）**（适配中） + + 同时支持 **Dense** 模型和 **混合专家（MoE）** 模型（MoE 适配中）。 +- **多种执行时间预测后端** — 可选： + - **AICB/AIOB** — 部分支持 DeepSeek-V3-671B、Qwen3-MoE-235B、Qwen3-Next-80B 的计算核与 TP、DP、PP、EP 通信量建模 + - **SimAI 仿真（Simulation）** — 基于 SimAI NS-3 的网络通信全栈仿真（支持 TP） + - **SimAI 解析（Analytical）** — SimAI 解析性能模型（支持 TP） + - **原版 Vidur [original]** — 支持 TP、DP、PP +- **负载生成与回放** — 支持真实 trace 回放，或使用固定/泊松分布生成合成请求。 +- **细粒度指标** — 记录： + - TTFT — 首 token 时延 + - TBT / TPOT — 相邻 token 时延 / 每输出 token 耗时 + - 端到端延迟 + - 通信开销 + - 计算开销 + - 调度延迟 + +--- + +## GPU 显存计算模块 + +本模块为现代 MoE（混合专家）模型的推理仿真提供精确的 GPU 显存估算，涵盖**模型参数显存**、**KV Cache 显存**以及 Prefill–Decode（PD）分离架构下的**最大批处理量**计算。 + +### 支持的注意力架构 + +| 架构 | 模型 | 说明 | +|---|---|---| +| **MLA**（多头潜在注意力） | DeepSeek-V3-671B | 使用 LoRA 压缩的 KV Cache（`kv_lora_rank` + `qk_rope_head_dim`），显著降低显存占用 | +| **MHA / GQA**（多头 / 分组查询注意力） | Qwen3-MoE-235B | 标准 KV Cache，每 token 每层使用 `num_kv_heads * head_dim` | +| **混合全注意力 + 线性注意力** | Qwen3-Next-80B | 每 4 层交替使用全注意力和线性（GDN）注意力 | + +### 核心组件 + +- **`ParamCounter`**（`vidur/utils/param_counter.py`）— 计算每层和每设备的参数量，支持 MLA、MHA/GQA、线性注意力和 MoE 专家权重，支持 FP8 量化。在 PD 分离架构下，根据 `prefill_world_size` / `decode_world_size` 分别返回 `(total_params, prefill_params, decode_params)` 三元组。 +- **`MemoryPlanner`**（`vidur/scheduler/utils/memory_planner.py`）— 规划 GPU 显存预算：`available = GPU_mem * (1 - margin) - param_mem`，计算 KV Cache 容量和最大并发请求数，包含 OOM 检测与建议输出。 +- **逐请求 KV Cache 追踪**（`vidur/entities/replica.py`）— 按请求粒度分配和释放 KV Cache 显存，支持运行时精确查询剩余容量。 + +### 参考与致谢 + +本 GPU 显存计算模块的开发参考了以下工作： + +- [InferSim](https://github.com/alibaba/InferSim) — 参数量计算与 KV Cache 估算方法论 +- [DeepSeek V3 Parameter Size Analysis](https://yangwenbo.com/articles/deepseek-v3-parameter-size.html) — DeepSeek V3 MLA 参数推导 +- [DeepSeek V3 参数推导详解](https://zhuanlan.zhihu.com/p/21455638257) — MLA 权重分解详细分析 + +衷心感谢以上资源为我们的实现提供了基础性的分析与指导。 + +--- + +## 支持的模型 + +- **DeepSeek-V3-671B**（SimAI PP/EP 通信及 GPU 显存管理模块适配中） +- **Qwen3-MoE-235B**、**Qwen3-Next-80B**（SimAI PP/EP 通信及 GPU 显存管理模块适配中） +- **meta-llama/Meta-Llama-3-8B** / **Meta-Llama-3-70B** +- **meta-llama/Llama-2-7b-hf** / **Llama-2-70b-hf** +- **codellama/CodeLlama-34b-Instruct-hf** +- **internlm/internlm-20b** +- **Qwen/Qwen-72B** + +--- + +## 📦 环境配置 + +### 1. 创建 Conda 环境 + +```bash +conda env create -p ./env -f ./environment.yml +``` + +### 2.（可选）更新开发依赖 + +```bash +conda env update -f environment-dev.yml +``` + +### 3. 激活环境 + +```bash +conda activate vidur +``` + +### 4. 安装 Python 依赖（使用阿里云 PyPI 镜像） + +```bash +pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ +pip install -r requirements-dev.txt -i https://mirrors.aliyun.com/pypi/simple/ +``` + +--- + +## ▶️ 运行示例 + +### 使用 AICB 运行 DeepSeek-671B + +**前置条件：** 需要 SimAI 和 AICB Docker 环境（参见 [README](../README.md) 了解搭建方法）。 + +完成环境配置后，运行以下命令： + +#### DeepSeek-671B + AICB（固定长度生成器） + +```bash +cd SimAI/vidur-alibabacloud + +python -m vidur.main --replica_config_pd_p2p_comm_bandwidth 800 \ + --replica_config_nvlink_bandwidth 1600 \ + --replica_config_rdma_bandwidth 800 \ + --replica_config_pd_p2p_comm_dtype float32 \ + --poisson_request_interval_generator_config_qps 100 \ + --synthetic_request_generator_config_num_requests 5 \ + --length_generator_config_type fixed \ + --fixed_request_length_generator_config_prefill_tokens 1024 \ + --fixed_request_length_generator_config_decode_tokens 10 \ + --trace_request_length_generator_config_trace_file ./data/processed_traces/splitwise_conv.csv \ + --cluster_config_num_replicas 4 \ + --replica_config_pd_node_ratio 0.5 \ + --global_scheduler_config_type split_wise \ + --replica_scheduler_config_type split_wise \ + --replica_config_model_name deepseek-671B \ + --replica_config_tensor_parallel_size 2 \ + --replica_config_num_pipeline_stages 1 \ + --replica_config_expert_model_parallel_size 8 \ + --random_forrest_execution_time_predictor_config_backend aicb +``` + +#### DeepSeek-671B + AICB（Trace 长度生成器） + +```bash +cd SimAI/vidur-alibabacloud + +python -m vidur.main \ + --replica_config_pd_p2p_comm_bandwidth 800 \ + --replica_config_nvlink_bandwidth 1600 \ + --replica_config_rdma_bandwidth 800 \ + --replica_config_pd_p2p_comm_dtype float32 \ + --poisson_request_interval_generator_config_qps 100 \ + --synthetic_request_generator_config_num_requests 10 \ + --length_generator_config_type trace \ + --trace_request_length_generator_config_max_tokens 1024 \ + --trace_request_length_generator_config_trace_file ./data/processed_traces/splitwise_conv.csv \ + --interval_generator_config_type poisson \ + --cluster_config_num_replicas 4 \ + --replica_config_pd_node_ratio 0.5 \ + --global_scheduler_config_type split_wise \ + --replica_scheduler_config_type split_wise \ + --replica_config_model_name deepseek-671B \ + --replica_config_tensor_parallel_size 2 \ + --replica_config_num_pipeline_stages 1 \ + --replica_config_expert_model_parallel_size 8 \ + --random_forrest_execution_time_predictor_config_backend aicb +``` + +> ✅ 完整参数说明可通过 `python -m vidur.main -h` 查看。 + +### 使用 SimAI 仿真运行 Llama-3-8B + +```bash +cd SimAI + +# 编译 SimAI-Simulation（ns3） +./scripts/build.sh -c ns3 + +# 生成网络拓扑（Spectrum-X_128g_8gps_100Gbps_A100） +python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py \ + -topo Spectrum-X -g 128 -gt A100 -bw 100Gbps -nvbw 2400Gbps + +cd SimAI/vidur-alibabacloud + +python -m vidur.main \ + --replica_config_pd_p2p_comm_bandwidth 800 \ + --replica_config_nvlink_bandwidth 1600 \ + --replica_config_rdma_bandwidth 800 \ + --replica_config_pd_p2p_comm_dtype float32 \ + --poisson_request_interval_generator_config_qps 100 \ + --synthetic_request_generator_config_num_requests 10 \ + --length_generator_config_type trace \ + --trace_request_length_generator_config_max_tokens 2048 \ + --trace_request_length_generator_config_trace_file ./data/processed_traces/splitwise_conv.csv \ + --interval_generator_config_type poisson \ + --cluster_config_num_replicas 4 \ + --replica_config_pd_node_ratio 0.5 \ + --global_scheduler_config_type split_wise \ + --replica_scheduler_config_type split_wise \ + --replica_config_model_name meta-llama/Meta-Llama-3-8B \ + --replica_config_tensor_parallel_size 4 \ + --replica_config_num_pipeline_stages 1 \ + --replica_config_expert_model_parallel_size 1 \ + --random_forrest_execution_time_predictor_config_backend simai_simulation \ + --random_forrest_execution_time_predictor_config_simai_dir ../ \ + --random_forrest_execution_time_predictor_config_simai_simulation_topo ../Spectrum-X_128g_8gps_100Gbps_A100 \ + --random_forrest_execution_time_predictor_config_simai_simulation_config ../astra-sim-alibabacloud/inputs/config/SimAI.conf +``` + +### 使用 SimAI 解析模型运行 Llama-3-8B + +```bash +cd SimAI + +# 编译 SimAI-Analytical +./scripts/build.sh -c analytical + +cd SimAI/vidur-alibabacloud + +python -m vidur.main \ + --replica_config_pd_p2p_comm_bandwidth 800 \ + --replica_config_nvlink_bandwidth 1600 \ + --replica_config_rdma_bandwidth 800 \ + --replica_config_pd_p2p_comm_dtype float32 \ + --poisson_request_interval_generator_config_qps 100 \ + --synthetic_request_generator_config_num_requests 10 \ + --length_generator_config_type trace \ + --trace_request_length_generator_config_max_tokens 2048 \ + --trace_request_length_generator_config_trace_file ./data/processed_traces/splitwise_conv.csv \ + --interval_generator_config_type poisson \ + --cluster_config_num_replicas 4 \ + --replica_config_pd_node_ratio 0.5 \ + --global_scheduler_config_type split_wise \ + --replica_scheduler_config_type split_wise \ + --replica_config_model_name meta-llama/Meta-Llama-3-8B \ + --replica_config_tensor_parallel_size 4 \ + --replica_config_num_pipeline_stages 1 \ + --replica_config_expert_model_parallel_size 1 \ + --random_forrest_execution_time_predictor_config_backend simai_analytical +``` + +### 使用原版 Vidur 运行 Llama-3-8B + +```bash +cd SimAI/vidur-alibabacloud + +python -m vidur.main \ + --replica_config_pd_p2p_comm_bandwidth 800 \ + --replica_config_nvlink_bandwidth 1600 \ + --replica_config_rdma_bandwidth 800 \ + --replica_config_pd_p2p_comm_dtype float32 \ + --poisson_request_interval_generator_config_qps 100 \ + --synthetic_request_generator_config_num_requests 10 \ + --length_generator_config_type trace \ + --trace_request_length_generator_config_max_tokens 2048 \ + --trace_request_length_generator_config_trace_file ./data/processed_traces/splitwise_conv.csv \ + --interval_generator_config_type poisson \ + --cluster_config_num_replicas 4 \ + --replica_config_pd_node_ratio 0.5 \ + --global_scheduler_config_type split_wise \ + --replica_scheduler_config_type split_wise \ + --replica_config_model_name meta-llama/Meta-Llama-3-8B \ + --replica_config_tensor_parallel_size 4 \ + --replica_config_num_pipeline_stages 1 \ + --replica_config_expert_model_parallel_size 1 \ + --random_forrest_execution_time_predictor_config_backend vidur +``` + +### 运行四场景套件 + +使用内置脚本快速验证所有支持的配置： + +```bash +bash examples/vidur-ali-scenarios/run_scenarios.sh --all +``` + +详细信息请运行 `bash examples/vidur-ali-scenarios/run_scenarios.sh --help`。 + +#### 四场景配置说明 + +以下场景已在 `run_scenarios.sh` 中预配置，所有场景共享下方硬件配置。 + +**共用硬件配置：** +- GPU：H20（h20_dgx），NVLink：1600 Gbps，RDMA：800 Gbps +- PD P2P 带宽：800 Gbps，数据类型：fp8 +- 请求生成：Poisson QPS=100，4 requests，固定 prefill=100 / decode=8 tokens + +| 场景 | 模型 | PD 分离 | World Size | TP | PP | EP | 全局调度器 | +|------|------|---------|------------|----|----|------------|------------| +| 1 | Qwen3-Next-80B (MoE) | 无 | 32 (dp=32) | 1 | 1 | 1（默认） | lor | +| 2 | Qwen3-Next-80B (MoE) | 是（P=2, D=6） | 8 | 1 | 1 | 1（默认） | split_wise | +| 3 | DeepSeek-671B (MoE) | 是（P=2, D=6） | 8 | 8 | 1 | 8 | split_wise | +| 4 | Qwen3-MoE-235B (MoE) | 是（P=2, D=6） | 8 | 4 | 1 | 4 | split_wise | + +> **说明：** 四个模型均使用混合专家（MoE）架构。EP 列反映脚本中 `--replica_config_expert_model_parallel_size` 的显式设定值；未显式指定时使用默认值 1。 + +#### 输出文件说明 + +**输出路径取决于运行方式：** + +- **`run_scenarios.sh`** --- 输出到 `examples/vidur-ali-scenarios/simulator_output/` +- **直接 `python -m vidur.main`** --- 输出到 `./simulator_output/`（或通过 `--metrics_config_output_dir` 指定的路径） + +每次运行产生如下目录： + +``` +