You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In the non-fused/reference path of Mamba2Simple, softplus is applied to dt twice — once explicitly before passing to the SSM step, and once inside the step function. This over-smooths the timestep values.
Fix
Removed the redundant softplus application to match the fused kernel behavior.
Sorry, can you describe the exact code path where the softplus is applied twice?
It seems you're referring to the path starting line 158? Are you claiming the softplus is applied on line 162 and again on line 182 (mamba_chunk_scan_combined)? But I don't think the latter applies the softplus by default.
mamba_chunk_scan_combined defaults to dt_softplus=False, so it receives the already-transformed dt and applies no further softplus. softplus is applied exactly once.
The proposed change moves the operation inside the Triton kernel via dt_softplus=True, which produces the same result. That may be a cleaner factoring, but there's no double application to fix in the current code.
Suggest re-labeling as a refactor/cleanup rather than a bug fix and close the PR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
In the non-fused/reference path of Mamba2Simple,
softplusis applied todttwice — once explicitly before passing to the SSM step, and once inside the step function. This over-smooths the timestep values.Fix
Removed the redundant softplus application to match the fused kernel behavior.