Modal deployment challenges with PyTorch 2.4+ and flash-attention compatibility

[testost]

Problem Description

Attempting to deploy H-Net on Modal Labs encounters C++ ABI compatibility issues between PyTorch 2.4+ (required for set_wrap_triton_enabled) and flash-attention builds.

Root Cause

H-Net requires PyTorch 2.4+ for the set_wrap_triton_enabled function, but pre-built flash-attention wheels are incompatible with newer PyTorch versions due to C++ ABI mismatches.

Error Details

/usr/local/lib/python3.10/site-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c105ErrorC2ENS_14SourceLocationESs

This indicates a mismatch between C++ standard library versions used to compile PyTorch vs flash-attention.

Attempted Solutions

1. Building flash-attention from source: Fails due to compiler environment issues (clang++ vs g++ mismatches)
2. CUDA version alignment: Tried matching CUDA 12.1 with PyTorch 2.4.1+cu121 but pip installs different versions
3. C++ ABI flags: Setting _GLIBCXX_USE_CXX11_ABI=0 during build still results in symbol errors
4. Various Python versions: Tested 3.10 and 3.11 with similar results

Working Configuration (Limited)

• PyTorch 2.3.1 + pip install flash-attn works
• But lacks set_wrap_triton_enabled required by H-Net

Environment

• Modal Labs container deployment
• CUDA 12.1.1-devel base image
• Python 3.10/3.11

Request

Could the H-Net team provide guidance on:

1. Recommended PyTorch/flash-attention version combinations
2. Alternative approaches for Triton operations if flash-attention compatibility is problematic
3. Docker/container-specific build instructions for Modal/similar platforms

This affects cloud deployment scenarios where pre-built wheels are preferred over source compilation.

[sukjunhwang]

Hi @testost ,

We have mostly used the combination of : Torch 2.5.1, and Triton 3.2.0.
But I also have encountered undefined symbol issue, and this usually happens when triton cache isn't removed. I believe the versions of Triton and Torch should not matter a lot as long as Flash Attention and Mamba kernels work properly, so could you try alternative versions of those packages?

[geranium12]

I updated the following libraries, and this resolved the problem for me:

"torch>=2.7.1",
"triton>=3.3.1",
"mamba-ssm>=2.2.5",
"flash-attn>=2.8.1",
"causal-conv1d>=1.5.2",

[jihwan1205]

I had the same error too.
Changing the flash-attn version to >= 2.8.1 simply solved the issue.
Hope it helps!

[malcolmsharpe]

For anyone who wants to run H-Net on Modal, I uploaded my quick-and-dirty scripts.

[jordiferrero]

What I did to make this work:

1. Use a prebuild pytorch AWS Image (e.g. Deep Learning OSS Nvidia Driver AMI GPU PyTorch 2.8 (Amazon Linux 2023) 20251103)
2. Manually install flash-attn-2.8.3 for the Python/pyTorch/CUDA version you already have installed using pre-built wheels
3. Git clone this repo and modify the pyproject.toml as @geranium12 suggested:

dependencies = [
    "torch>=2.7.1",
    "triton>=3.3.1",
    "mamba-ssm>=2.2.5",
    "flash-attn>=2.8.1",
    "causal-conv1d>=1.5.2",
    "einops",
    "optree",
    "regex",
    "omegaconf",
]

4. Just cd to cloned folder and pip install -e .