Implement SDL3 Audio backend

[hwsmm]

#6002 (comment)

[smoogipoo]

This is super cool, though I don't see osu! using this in the near future if ever. I'd see this implemented as an additive nuget package, and exposed through some way that isn't the framework config. For example, it could be included in HostOptions or simply as a virtual method in Game.

[peppy]

I'd actually like to experiment with this and see how good support is for the upcoming changes I want to make with WASAPI initialisation. So I don't want to throw this out. Moving away from bass would be a huge consideration, but I wouldn't throw it away.

I'm going to mark this as a draft as I don't see it getting reviewed or merged anytime soon, but I still useful to have around as a reference for what is involved in making this work, and potential performance / latency cross-checking in a future.

[hwsmm]

I'll maintain this until some of you have anything to do with this, mostly because I am now so used to playing the game with these patches, I can't really go back to BASS...
FWIW, on Linux with Pipewire at very small buffer, hitsound latency was almost on par with patched wine which osu! linux players mostly use. It produces some artifacts, so shouldn't be default, though.

[peppy]

@hwsmm thanks for keeping this up-to-date. curious on your thoughts on the move to sdl3 – does this help in any way on the audio side of things? i don't want to promise anything but we have internally discussed the potential of moving to sdl3 as an audio provider at some point.

curious on how wasapi is looking specifically, as i read that sdl3 was going to add better support, potentially replacing my attempt at initialising for lower latency (#6088) which ended up having compatibility issues and limited results.

[hwsmm]

SDL3 Audio doesn't give any improvements over SDL2 for the usecase of osu!framework in my opinion.
From what I see, improvements in SDL3 audio is mainly to let developers reduce burden of writing audio subsystems by themselves, and my implementation already has mixers and everything, so they are kind of meaningless here and even add a bit of overhead in some cases.

SDL has a good WASAPI implementation, but they don't provide any means to adjust buffer sizes (https://github.com/libsdl-org/SDL/blob/main/src/audio/wasapi/SDL_wasapi.c#L635 -- only uses default '0' value).
Buffer length usually ends up being around 10ms (or more, should be plenty good?) on most systems I tested on. I am not sure how better it could be compared to BASS DirectSound.
We can make use of this API to determine lowest possible latency, but it's only available on Windows 10 or higher. I can probably write a PR (or a patch for our SDL builds?) to SDL for this if anyone wants, but I guess you can also just implement it within C# and pass the value to BASSWASAPI.
Though I didn't know such API existed until now. I don't really play osu! on Windows, anyway..

However, the problem that hinders real low latency lies in .NET GC overhead. As I wrote in the PR description and the previous comment, audio stutters a lot if I force low latency on Linux, and it should be the same on Windows, too.
Any value below 10ms seems to cause underruns in audio, and it's very noticeable if you go back to main menu from song select in game.
I remember that the same happened in your WASAPI testing discussions on Discord, and this is probably because WASAPI callback exists in managed land.
I even wrote a simple C library out of boredom to handle SDL audio callback in unmanaged land, and it solved most of audio stutters issues on very small buffer on Linux (around 1ms), but you won't want this since it requires a couple of additional native libraries.

So what I am trying to say here is, that even with the Windows API above, it's going to be pretty hard to get the stable low latency audio experience unless audio is handled in completely unmanaged land.
The best solution might be making a native library for BASSWASAPI callback if you only consider Windows, but I would like to see any kind of support for Linux since there are some players (including me) not wanting to make a complete switch to lazer because patched WINE provides better latency.

---

... if you ask me what's the point of this PR then:

1. Better support for Linux audio servers
2. Audio is at least fine in gameplay at very low latency, it's not so good in song select, though.. (Not saying that audio is always bad, it sounds nice around 10ms)
3. Can get rid of BASS, a closed source library, without introducing new native libraries except for libswresample, which is a part of ffmpeg

[smoogipoo]

However, the problem that hinders real low latency lies in .NET GC overhead.

That's, honestly, the first time I'm hearing of this. I would imagine that code returns in a fraction of a microsecond so I wouldn't expect the GC to have that sort of impact on it, but I suppose it would make sense for super small buffer lengths.

If it is GC and not e.g. the cost of context switching, there's a few things I'd try:

• Wrapping in GC.TryStartNoGCRegion()/GC.EndNoGCRegion()?
• Using UnmanagedCallersOnly.
• Using SuppressGCTransition.

Perhaps, all at the same time.

[hwsmm]

I guess my comment was a bit ambiguous. I meant that the audio callback returns fast, but the problem is supposedly 'stop-the-world' effect of GC.
Whenever STW occurs, SDL fails to get audio data in time from the audio callback located in managed land.
Please correct me if I misunderstood your message, I have only used C# for less than an year at this point.

Here are my test results for things you mentioned:

1. GC.TryStartNoGCRegion()/GC.EndNoGCRegion() pair mostly fixes the problem, except for longer stutter from time to time.
2. UnmanagedCallersOnly is already applied to the audio callback.
3. SuppressGCTransition doesn't seem to help. Only one native method is used in the callback, anyway..

[TheComputerGuy96]

It would be nice to combine all of this audio component work and make a new open-source BASS-compatible audio library
that everyone could use (not just C# applications) in a separate repository (I'm personally going to call it Treble until a better name can be found)

[hwsmm]

First of all, this PR is not compatible with BASS. It just implements o!f audio components in C# and SDL, while the current o!f audio backend is written in BASS API.
Which means, this PR calculates current track position by converting data position to milliseconds in C#, but current o!f calls BASS_ChannelGetPosition (and some more) to do the same.
It would be very time consuming to re-expose internal APIs in a new project and to make it compatible with BASS and its addons (o!f uses BASSMIX, BASSFX and BASSWASAPI), and BASS is still maintained well.

I do love open source, and it's true that I wrote this PR to get rid of a closed source lib from osu!framework, but there is only little point to both audio library consumers and us in maintaining a new audio library when there are mature open-source audio libraries out there. Better contribute to them if they are missing some features. It could be another example for xkcd new standard in my opinion.

Just consider this as a random audio backend for a game engine. Unity, Unreal and Godot, they all have their own audio backend, but they are not standalone, right?

[smoogipoo]

Am thinking you might even be able to make a C# NativeAOT project for it if people are uncomfortable with C. I have a bunch of experience setting that up on all relevant platforms including mobile if needed.

[hwsmm]

Sorry for not responding quickly.

That's actually not surprising since I started writing this backend because I was consistently hitting early on the game no matter what offset I use.

Honestly, I don't really know what to do, so I couldn't reply to your comment on time. I'm fine with maintaining either C or C# version, but if we end up with C# one, I think we'd get another bug report about audio stutters.
The native library unfortunately contains most of core logic to avoid GC stutters as much as possible. It is just an audio library. You create a channel, add it to a mixer, and then it plays.

I guess NativeAOT might work if people dislike using C, but we are not compiling the entire game with it, right? If we are compiling the audio framework in a shared object with NativeAOT, should we make bindings for it, or is there a way to dynamically load a natively AOT-compiled library in C#?

[smoogipoo]

I'm fine with maintaining either C or C# version, but if we end up with C# one, I think we'd get another bug report about audio stutters

The hopeful idea is for the NativeAOT lib to do things in such a way that every allocation is accounted for, while also making it touchable by core developers (who are most experienced in C#).

I guess NativeAOT might work if people dislike using C, but we are not compiling the entire game with it, right?

Nah. I don't think we ever will - the JIT is actually quite useful for us optimisation-wise. But that won't help in this case because the idea is to isolate the osu! GC from the NativeAOT GC.

If we are compiling the audio framework in a shared object with NativeAOT, should we make bindings for it, or is there a way to dynamically load a natively AOT-compiled library in C#?

You need to expose functions using [UnmanagedCallersOnly] (EntryPoint is required) (example). Then you use it just as a normal native lib (example).

[hwsmm]

Yeah, I agree on the point, too.

But I am worrying about how we should pass objects between .NET and NativeAOT. Do we need to create ObjectHandles for everything, as .NET objects are not blittable?

[smoogipoo]

Yeah. You'd basically interface with it as if it's a normal C lib - there's no passing of objects or hackily dereferencing into a matching managed signature (except for blittable/unmanaged types ofc).

[hwsmm]

Well, it sounds like rewriting my C library into C# again if I am understanding it correctly.

I think it shouldn't take so long since I already have the .NET part from my native project (unless I run into weird issues). I'll start writing it sooner or later if you are fine with it.

[smoogipoo]

Hold off on that one for a bit. Do you have a sample of the C lib to look at to see what needs to be done?

[hwsmm]

https://gist.github.com/hwsmm/c2bb0e55694e14c83372b8bdcb73830b

Here are (hand-written) function bindings. Once all of them are implemented, it should work well.
I hope function names are clear enough to tell what they do.

It doesn't have fancy things like SDLBool because I am the only consumer of this library...

[smoogipoo]

I was hoping to see the implementation because people may start to get uncomfortable if the library is very large. In general I think people would feel comfortable as long as most implementation is still managed and the native part is only really fundamental processing.

[hwsmm]

Oh, if that's the reason, I can send you the compressed source tree. I don't want it to be public if it is not eventually going to be released, also it needs a lot of polishing so...

Can I send you a mail to the address in your GitHub profile?

However, I am not sure about the border of fundamental processing, though. The native part should know all the information that it needs to play audio without relying on C# part to avoid stutters, which includes volume, balance, tempo, frequency and more.

At best, 'Most implementation' can only be about exposing information (duration, channel counts, current time and so on) to the game, telling native channels to play/stop, and creating/destroying native channels when needed.

[smoogipoo]

Can I send you a mail to the address in your GitHub profile?

Yep, that's fine. I'll have a look at it with peppy and see if it's something we're comfortable with the scope/having it.

[hwsmm]

I've just sent a mail!

Forgot to add in the mail. I am planning to implement decoder in .NET side, because it adds a bit of complexity and may cause link problems with FFmpeg.

[hwsmm]

Starting from https://discord.com/channels/188630481301012481/589331078574112768/1396539637861847141, I copied most part from this PR to the new library, and I got most things working... on my PC.

As a side note, I was actually a few hours before (pre-)releasing my native library here when smoogi sent me that message since it had been pretty stable lately, so I was not motivated to write it again in NativeAOT, but I ended up deciding to do because it required much less effort because I already had some in C#.

I honestly got surprised with Satori. When I finished the basic implementation and tested with plain NativeAOT, the result was not good on very small buffer (around 1ms).

However, with Satori, it still had some underruns, but was very acceptable than plain NativeAOT and this PR.

Anyway, here are some points to know:

• It is LGPL because SoundTouch.Net (LGPL v2.1) is statically compiled.
• BASS is the default decoder because it needs to work for now. It can also use FFmpeg 7 with libswresample and audio decoder if it is available and BASS init code is removed from SDL3AudioManager.cs.
• Decoder is integrated because I want to keep API stable for my another native library (not public yet).
• Android probably works (not tested yet), but no iOS because I don't have any Apple devices. Too cumbersome to figure out xcframework when I can only use CI for Apple toolchain.
• Satori is not available on macOS x86_64 (due to build failure - this upstream issue suggests that the issue has been fixed in new XCode 16 releases but no luck for me) and Android (don't know how)
• Repository structure is pretty rough right now. I may fix in the future, but it works for now (repo).
• You can set an environment variable SDL_AUDIO_DEVICE_SAMPLE_FRAMES to adjust latency. 128 should work well on Windows with a good audio driver.

How to try:

1. git clone -b sdl-audio https://github.com/hwsmm/osu-framework.git
2. Apply this commit on top of upstream osu
3. cd osu && ./UseLocalFramework.sh
4. cd osu.Desktop && dotnet run
5. Edit AudioDriver in framework.ini to SDL3

This thing is still very early, so it may be (completely) broken in some cases. I'd be grateful if you can try and let me know if it works well.

[hwsmm]

@smoogipoo I am not expecting an immediate response, but here is a mention in case you didn't enable notification on this thread.

[smoogipoo]

This is still in my emails but very low priority at the moment. I've been waiting on a conclusion to the discussion in #705 + #6628 because @peppy's already reviewing that.

[Rudicito]

@hwsmm I tested your changes. Note that I'm using 2025.816.0-lazer, not the latest version, because your framework is a bit outdated.

I'm on Linux (Fedora 42).

I tried with this beatmap. Note that I’ve set a 10 ms audio offset, since that’s what I use in my regular lazer client, that I deactivate beatmap hitsound, and that I use argon pro.

With BASS:

With SDL3 (without set SDL_AUDIO_DEVICE_SAMPLE_FRAMES):

With SDL3 (SDL_AUDIO_DEVICE_SAMPLE_FRAMES = 128):

With SDL3 (SDL_AUDIO_DEVICE_SAMPLE_FRAMES = 64)

For SDL3 and SDL_AUDIO_DEVICE_SAMPLE_FRAMES set to 64 and 128, the hitsound feels more responsive, feel like I was often clicking early. Didn't encounter audio issues with these values. In the editor, I feel like the hitsound are early.

With SDL_AUDIO_DEVICE_SAMPLE_FRAMES set to 32, sound start crackling, didn't try playing a beatmap.

Hope this help, you can ask me to test things if you want.

Maybe I should have test with no audio offset? I can do it if you want.

[hwsmm]

@Rudicito Hey, thanks for testings! Reporting issues you encounter while using it is sufficient for now (you may use whatever offset you want). sdl-audio-alt branch is now updated, so you can probably update your game.

[unknowwiiplayer]

The following was written with AI assistance based on my own testing and findings.

I am not an audio developer, so I cannot evaluate the architectural or maintenance concerns involved in merging this work. My contribution here is empirical: I am an experienced taiko player, and input-to-hitsound latency is the main reason I normally avoid osu! and prefer OpenTaiko with ASIO on Windows.

I tested the newer sdl-audio-alt work against osu! 2026.711.0-lazer on CachyOS with PipeWire.

To get the current game compiling, I used:

• the sdl-audio-alt branch of hwsmm/osu-framework
• the existing AudioFilter compatibility patch
• a small additional adaptation for the newer effects in AfToggleSection.cs

On the osu! side, the additional adaptation was small: six newer AddEffect() / RemoveEffect() calls were migrated to GetNewEffect(), Apply(), and Remove().

The result compiled successfully.

I was then able to play stably with PipeWire running at a 48-frame quantum at 44.1 kHz.

The gameplay difference was not subtle. Input-to-hitsound response felt dramatically tighter, and osu! felt better to me than it ever has before, including my experience playing on Windows. I only play taiko, so that is the mode I can confidently evaluate.

I understand that a working local build does not address all of the architectural, maintenance, mobile, or cross-platform concerns involved in upstreaming an alternative audio backend.

However, it does demonstrate that the newer SDL3 audio work can still run on the current game, and that its practical benefit for latency-sensitive Linux players can be enormous.

I would really like to see this work revisited and given a concrete path toward an officially supported low-latency audio option, even if it initially remains experimental or Linux-only.

I can provide the exact osu! diff and testing details if they are useful.

[hwsmm]

I am sorry, but osu! project does not accept any kind of AI assistance (including documentation things). Please take a look at this section: https://github.com/ppy/osu-framework#contributing

I just updated my patches in the sdl-audio-alt comment. It's not a big deal anyway, since I still play the game with my backend and fix issues as I find them.

Also I moved the old branch to sdl-audio-legacy, since I have not been touching it for long time.