Discussion: SIMD strategy for pandas C/C++ code

[jbrockmendel]

Current state

We don't have any direct SIMD code ourselves at the moment, but there are several PRs that want to add some or are experimenting with it:

• PERF: Use SIMD for read_csv C tokenizer #64515 uses hand-written SSE2 (x86-64) and NEON (arm64) intrinsics in the C tokenizer to scan for special characters 16 bytes at a time
• PERF: [POC] compute skew and kurtosis with SIMD Using Vector Extensions #64582
• PERF: Add SIMD instructions with xsimd to reduce moments #64905

Some background

• Distributing a package containing SIMD code: https://pypackaging-native.github.io/key-issues/simd_support/
• NumPy NEPs on the topic: https://numpy.org/neps/nep-0038-SIMD-optimizations.html, https://numpy.org/neps/nep-0054-simd-cpp-highway.html (

---

Original content:

Everything below this line is Claude, opened upon request in #64515.

[Deleted per Joris's comment]

[Alvaro-Kothe]

Thanks for starting the discussion @jbrockmendel.

Just a disclaimer, I am not an expert on the topic.

The core goal is to leverage CPU-specific vector extensions (SIMD) to accelerate performance. The main challenge is the hardware diversity:

• For x86, we have a hierarchy of extensions (AVX-512, AVX2, AVX, SSE4, etc.). For ARM, the primary extension is NEON.
• Since we cannot guarantee the host CPU supports a specific instruction set, we must utilize runtime dispatching. This involves compiling multiple versions of the same function and selecting the most optimized version supported by the host CPU at runtime.
• On x86_64, the extensions are backward compatible (e.g., a CPU supporting AVX-512 will also support AVX2 and SSE). So, it's possible to select a few number of targets to reduce compilation time and binary size.

---

For the options provided by @jbrockmendel

• Hand-written intrinsics:
  • Pros: Indeed it doesn't add any new dependencies.
  • Cons: It will be a headache to maintain, because we would need to manually implement and test unique versions for every extension we want to target.
• Compiler vector extensions:
  • Pros: Simplifies implementation; allows the compiler to handle multi-versioning and runtime dispatch via directives.
  • Cons: Not available on MSVC, which in my opinion makes it unviable.
• Meson SIMD module:
  • This is a tool that may be used with others, it's not a contender. It is best used in conjunction with hand-written intrinsics to manage the compilation of different instruction sets.
• xsimd:
  • It's a header-only library, it would be integrated similarly to fast_float.
  • pros: provides an abstraction layer over different architectures. It allows for writing architecture-independent code and includes built-in runtime dispatching.
  • cons: In my experience, it is quite heavy on tooling. It spiked clangd memory usage making it unusable for me.
• Highway: Can't opine; didn't try it.

---

It looks like they are also integrating SIMD directly on Rust and C++. The C++ case seems to be unreachable for now, because it's experimental and Rust seems hard to integrate here.

[jorisvandenbossche]

Since we cannot guarantee the host CPU supports a specific instruction set, we must utilize runtime dispatching. This involves compiling multiple versions of the same function and selecting the most optimized version supported by the host CPU at runtime.

That is not entirely true if we limit ourselves (initially) to a base set of instruction sets. As far as I understand, it should be safe (e.g. for packaging wheels) to assume that certain instruction sets are always available, and we can thus just compile a single version (still choosing the right set depending on the architecture of course, but entirely compile-time, without runtime logic).

This might make the discussion easier if we first tackle that question for the short-term, and leave the more complex questions about compiling for multiple microarchitectures and runtime dispatch for a second step.

For a baseline, I think it should definitely be safe to assume x86-64-v1 for x86-64 (which includes SSE2) and NEON for any of the arm64 we support (Linux / macOS (Apple Silicon) / Windows ARM). Checking other packages:

• NumPy has been assuming a baseline of SSE3, but recently (in numpy 2.4) switched to x86-64-v2 (which goes up to SSE4_2), see https://numpy.org/devdocs/release/2.4.0-notes.html#modulate-dispatched-x86-cpu-features (numpy/numpy#27851 / numpy/numpy#28896).
• Arrow C++ has a default SIMD level of SSE4_2, and I don't see this default overridden in the builds, so I assume the pyarrow wheels have that level (so essentially also x86-64-v2)
• Scipy probably has at least SSE2 of SSE3, but it is a bit unclear. There is an issue about this here

If we limit it initially to a baseline, then the question reduces to: how to write the SIMD-specific code? Use hand-written code per instruction set (which would at least be for x86-64-v1/2 and ARM NEON), or use some abstraction library (like xsimd or highway)?

Personally, I am fine to start with some hand-written intrinsincs (like @jbrockmendel is doing in #64515, if this is only for limited use cases. But then I would also prefer to stick to a single approach for now (i.e. align the different PRs we have to use a similar way of doing things).
But when using this more frequently, I think we should still look into one of the abstractions libraries to make this more maintainable.

Some other additional questions to discuss:

• Do we need a build flag to control this? Do we need to allow to disable optimizations entirely? (if initially sticking to a baseline, this might be less important, however numpy does allow to even disable the baseline AFAIU)
• Do we need to add a test build with the optimizations disabled to ensure coverage for that case? (i.e. do we officially support this?)
• We should probably add Windows ARM to the normal CI, as currently that seems to be only tested in the wheels build?

[jbrockmendel]

1. "We should use consistent patterns" is worthwhile
2. I don't start to think about standardizing patterns until we have 3+ usages, and since we're currently at "2, neither of them merged", I view this more as a "do what it takes to get the team on board" issue than a "decide on the right policy" issue.
3. Claude estimates wheel size impact of 50-200kb for xsimd, 200-500kb for highway (plus some per-function cost). I'd like to confirm those numbers.
4. The embarrassing truth is that I'm relying on claude for most of this topic/PRs. It is better than me at this and will only get more so over time. By the same token, it will get more reliable and the cost of maintaining this code will go down. I see this as a reason to get more aggressive in optimization over time.
5. The only part I really feel strongly about is that we shouldn't be leaving perf on the table for these super-commonly-used functions.

[Alvaro-Kothe]

• Do we need a build flag to control this? Do we need to allow to disable optimizations entirely?

+1 on adding an option to disable SIMD optimizations. Some reasons to disable SIMD include:

• Reduce binary size.
• Reproducible results involving floating point arithmetic, as SIMD changes the order that the values are accumulated in reductions.

• Do we need to add a test build with the optimizations disabled to ensure coverage for that case?

I think it's important to ensure correctness on platforms that we don't support vectorization, but I think that the Pyodide job, that compiles for web assembly, will test the non-vectorized case (But need to verify).

• We should probably add Windows ARM to the normal CI, as currently that seems to be only tested in the wheels build?

Querying cpu capabilities (at least for x86) is quite different between MSVC and GCC/Clang that it's probably worth to have at least one job to check MSVC+arm.

[jorisvandenbossche]

4. The embarrassing truth is that I'm relying on claude for most of this topic/PRs. It is better than me at this and will only get more so over time.

FWIW if you want to push forward this topic, I would recommend doing some research of your own (which can still be AI-assisted). While I trust that Claude or other agents can write simd code quite reliably, I wouldn't rely on it too much for the high-level questions at the moment. As a data point: I have been inclined to just delete or hide your top post, because your Claude-generated summary contains enough incorrect and irrelevant items to make it rather distracting, and essentially useless as a basis for discussing this.

---

@jbrockmendel something that might be useful for the discussion, is if you could check if #64515 would be possible to do using one of the abstraction libraries (eg I have no idea if the specific operation you are using there is something that fits their abstractions)
(just the compile-time usage I mean, to be able to write a single version of the simd code, not with dynamic dispatch etc)

I think it's important to ensure correctness on platforms that we don't support vectorization, but I think that the Pyodide job, that compiles for web assembly, will test the non-vectorized case (But need to verify).

WebAssembly also does support simd instructions AFAIK, but no idea if those would currently get enabled in the pyodide build (so at least long term not a good option to rely on for testing non-vectorized code path)

[Alvaro-Kothe]

WebAssembly also does support simd instructions AFAIK, but no idea if those would currently get enabled in the pyodide build

Unlike other architectures in CI that have a minimum SIMD instruction set enabled by default, Emscripted (used to compile to WASM) requires an explicit opt-in. To enable SIMD in Pyodide, it's necessary to compile with -msimd128.

For those interested, the Emscripten SIMD documentation provides a good overview on how to write, compile and available instructions.

[jan-wassenberg]

Highway TL here. I'd strongly advise using an abstraction to make the code more readable and avoid duplication. More and more instruction sets are also coming (SVE, RISC-V etc). x86 intrinsics are easy to translate to Highway.
Happy to advise.

[jbrockmendel]

On last week's dev call I expressed an opinion "I'm happy to go with whatever Joris prefers if it'll let us move forward on #64515, #64582". Joris expressed frustration with me for not having a Real Opinion. After looking at #65352, I'm ready to offer a Real Opinion of "If it'll avoid us having to duplicate code the way 65352 does, then I am in favor of using one of these libraries. Still no opinion on which"

[jan-wassenberg]

:)
I am of course biased, but happy with what we've built.
One neutral perspective is https://www.mnm-team.org/pub/Fopras/rock23/.