Skip to content

Add model configs: GLM, DeepSeek V4, Inkling, Laguna, Laguna NVFP4 #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rounakbende10 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add model configs: GLM, DeepSeek V4, Inkling, Laguna, Laguna NVFP4 #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions src/coding_agent_bench/models/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@
RedHatAI_Mistral_Small_4_119B_2603_NVFP4,
RedHatAI_NVIDIA_Nemotron_3_Super_120B_A12B_NVFP4,
RedHatAI_Qwen3_6_27B_FP8,
RedHatAI_GLM_5_2_FP8,
RedHatAI_DeepSeek_V4_Flash,
RedHatAI_DeepSeek_V4_Flash_NVFP4_FP8,
RedHatAI_Inkling_Small,
RedHatAI_Laguna_S_2_1,
poolside_Laguna_S_2_1_NVFP4,
)

MODEL_CONFIGS: list[type[ModelConfig]] = [
Expand All @@ -15,6 +21,12 @@
RedHatAI_Mistral_Small_4_119B_2603_NVFP4,
RedHatAI_NVIDIA_Nemotron_3_Super_120B_A12B_NVFP4,
RedHatAI_Qwen3_6_27B_FP8,
RedHatAI_GLM_5_2_FP8,
RedHatAI_DeepSeek_V4_Flash,
RedHatAI_DeepSeek_V4_Flash_NVFP4_FP8,
RedHatAI_Inkling_Small,
RedHatAI_Laguna_S_2_1,
poolside_Laguna_S_2_1_NVFP4,
]

MODEL_REGISTRY: dict[str, ModelConfig] = {cls.name: cls() for cls in MODEL_CONFIGS}
Expand Down
115 changes: 114 additions & 1 deletion src/coding_agent_bench/models/configs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ class RedHatAI_NVIDIA_Nemotron_3_Super_120B_A12B_NVFP4(ModelConfig):
]

class RedHatAI_Qwen3_6_27B_FP8(ModelConfig):

name = "RedHatAI/Qwen3.6-27B-FP8"
model_max_len = 131072
Comment thread
taagarwa-rh marked this conversation as resolved.
args = [
Expand All @@ -90,3 +90,116 @@ class RedHatAI_Qwen3_6_27B_FP8(ModelConfig):
"--tool-call-parser", "qwen3_coder",
"--default-chat-template-kwargs", '{"enable_thinking": true}',
]


class RedHatAI_GLM_5_2_FP8(ModelConfig):
# Verified: 8x H200 141GB, concurrency 2.23x at 262K context

@taagarwa-rh taagarwa-rh Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is enough concurrency for us to consider using vLLM over OpenRouter for GLM 5.2. Let's keep it in the PR though for informational purposes. How does it do on 8x B200?

# Note: cannot fit 1M context on 8x H200 (needs 52.68 GiB KV, only 23.78 GiB available)

name = "RedHatAI/GLM-5.2-FP8"
model_max_len = 262144
args = [
"--model", "RedHatAI/GLM-5.2-FP8",
"--dtype", "auto",
"--max-model-len", "262144",
"--trust-remote-code",
Comment thread
taagarwa-rh marked this conversation as resolved.
"--kv-cache-dtype", "fp8",
"--enable-expert-parallel",
"--enable-auto-tool-choice",
"--reasoning-parser", "glm45",
"--tool-call-parser", "glm47",
]


class RedHatAI_DeepSeek_V4_Flash(ModelConfig):
# Verified: 8x H200 141GB, concurrency 9.91x at 1M context
# Note: --moe-backend deep_gemm_mega_moe is B200-only (SM100)

name = "RedHatAI/DeepSeek-V4-Flash"
image = "vllm/vllm-openai:v0.27.1"
model_max_len = 1048576
args = [
"--model", "RedHatAI/DeepSeek-V4-Flash",
"--dtype", "auto",
"--trust-remote-code",
"--kv-cache-dtype", "fp8",
"--block-size", "256",
"--enable-expert-parallel",
"--enable-auto-tool-choice",
"--tokenizer-mode", "deepseek_v4",
"--tool-call-parser", "deepseek_v4",
"--reasoning-parser", "deepseek_v4",

@taagarwa-rh taagarwa-rh Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash mentions a couple flags I don't see here:

  --attention_config.use_fp4_indexer_cache True \
  --moe-backend deep_gemm_mega_mo

Did you test these and find they are not needed?

@rounakbende10 rounakbende10 Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--moe-backend deep_gemm_mega_moe is B200-only (SM100) crashes on H200 (SM90). We tested on H200 initially, so excluded it. On B200 we did test with mega_moe and got 12.81x vs 9.91x on H200 without it. We can add it as a comment noting it's B200-only, or add a separate B200 config with it enabled. what do you think?

--attention_config.use_fp4_indexer_cache True is used only for NVFP4 variant. I will confirm once again with this flag if theres any change in concurrency

@taagarwa-rh taagarwa-rh Sep 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh that is something I hadn't considered. Should we add a per-hardware args section to the ModelConfigs? E.g.

from coding_agent_bench.nebius_utils import B200, B200x8

class RedHatAI_DeepSeek_V4_Flash(ModelConfig):
    ...
    hardware_extra_args: dict[str, list[str]] = {
        B200.name: ["--moe-backend", "deep_gemm_mega_moe"],
        B200x8.name: ["--moe-backend", "deep_gemm_mega_moe"],
    }
    ...

Then somewhere when building the vLLM command it can reference

args += model_config.hardware_extra_args.get(hardware.name, [])

Regarding --attention_config.use_fp4_indexer_cache True, I'm seeing that listed on the FP8 model as well, so I think it's needed for both oh that's only on the B200 as well. That would be another one to add to the B200 specific args

@rounakbende10 rounakbende10 Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on the hardware_extra_args approach. start_model already extracts the GPU preset from the instance so we can match on that. I'll implement it. DeepSeek gets --moe-backend deep_gemm_mega_moe on B200s and --attention_config.use_fp4_indexer_cache True

]


class RedHatAI_DeepSeek_V4_Flash_NVFP4_FP8(ModelConfig):
# Verified: 8x H200 141GB, concurrency 9.81x at 1M context
# Note: Marlin FP4 fallback on H200 (no native SM100 FP4), similar concurrency to base

name = "RedHatAI/DeepSeek-V4-Flash-NVFP4-FP8"
image = "vllm/vllm-openai:v0.27.1"
model_max_len = 1048576
args = [
"--model", "RedHatAI/DeepSeek-V4-Flash-NVFP4-FP8",
"--dtype", "auto",
"--trust-remote-code",
"--kv-cache-dtype", "fp8",
"--block-size", "256",
"--enable-expert-parallel",
"--enable-auto-tool-choice",
"--tokenizer-mode", "deepseek_v4",
"--tool-call-parser", "deepseek_v4",
"--reasoning-parser", "deepseek_v4",

@taagarwa-rh taagarwa-rh Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment above

]


class RedHatAI_Inkling_Small(ModelConfig):
# Verified: 8x H200 141GB, BF16, concurrency 13.22x at 1M context

name = "RedHatAI/Inkling-Small"
image = "vllm/vllm-openai:v0.27.1"
model_max_len = 1048576
args = [
"--model", "RedHatAI/Inkling-Small",
"--dtype", "auto",
"--trust-remote-code",
"--kv-cache-dtype", "fp8",
"--enable-auto-tool-choice",
"--tool-call-parser", "inkling",
"--reasoning-parser", "inkling",

@taagarwa-rh taagarwa-rh Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://recipes.vllm.ai/thinkingmachines/Inkling-Small?variant=bf16 has a couple flags I don't see here:

  --tokenizer-mode inkling \
  --kernel-config.enable_flashinfer_autotune=False \

Did you test these and find they are not needed?

]


class RedHatAI_Laguna_S_2_1(ModelConfig):
# Verified: 8x H200 141GB, BF16, max-model-len 1048576, concurrency 30.61x

name = "RedHatAI/Laguna-S-2.1"
model_max_len = 1048576
args = [
"--model", "RedHatAI/Laguna-S-2.1",
"--dtype", "auto",
"--trust-remote-code",
"--kv-cache-dtype", "fp8",
"--enable-auto-tool-choice",
"--reasoning-parser", "poolside_v1",
"--tool-call-parser", "poolside_v1",
"--default-chat-template-kwargs", '{"enable_thinking": true}',
]

class poolside_Laguna_S_2_1_NVFP4(ModelConfig):
# Verified: 1x B200 183GB, NVFP4, max-model-len 1048576, concurrency 2.33x

@taagarwa-rh taagarwa-rh Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for testing this, same as above looks like this won't be enough concurrency so we'll prefer the BF16 version you added. No changes needed here. Thanks again for trying this, this will serve as justification for using the larger node to run this model


name = "poolside/Laguna-S-2.1-NVFP4"
model_max_len = 1048576
args = [
"--model", "poolside/Laguna-S-2.1-NVFP4",
"--dtype", "auto",
"--trust-remote-code",
"--kv-cache-dtype", "fp8",
"--enable-auto-tool-choice",
"--reasoning-parser", "poolside_v1",
"--tool-call-parser", "poolside_v1",
"--default-chat-template-kwargs", '{"enable_thinking": true}',
]