Skip to content

Optimize swizzle_dyn on ssse3 - #543

Merged
programmerjake merged 1 commit into
rust-lang:masterfrom
Shnatsel:optimize-ssse3-swizzle-dyn
Jul 26, 2026
Merged

programmerjake merged 1 commit into
rust-lang:masterfrom
Shnatsel:optimize-ssse3-swizzle-dyn

Conversation

@Shnatsel

@Shnatsel Shnatsel commented Jul 25, 2026

Copy link
Copy Markdown
Member

The idea was suggested by @axnsan12.

Changes the control sequence from vpcmpgtb + vpblendvb to vpaddb + vpor. 15% throughput improvement on Haswell, 20% improvement on skylake, zen1, zen3. For the 256-bit case, 38% improvement on haswell, 12% on skylake, zen1 and zen3 unchanged. Disassembly measured on fealrless_simd similarly to #542.

Expands test coverage to cover byte values 128-255 which would misbehave without idxs | zeroing_bits.

…e correctness for mask values 128-255. Changes the control sequence from vpcmpgtb + vpblendvb to vpaddb + vpor. 15% throughput improvement on Haswell, 20% improvement on skylake, zen1, zen3. For the 256-bit case, 38% improvement on haswell, 12% on skylake, zen1 and zen3 unchanged. Expand test coverage to cover byte values 128-255 which would misbehave without the OR.
@programmerjake
programmerjake merged commit 832ff3c into rust-lang:master Jul 26, 2026
53 checks passed
@SuperSamus

Copy link
Copy Markdown

Couldn't this version of zeroing_idxs also be used on the AVX2 shuffle like this?

        let lolo = avx2_cross_shuffle::<0x00>(bytes.into(), bytes.into());
        let hihi = avx2_cross_shuffle::<0x11>(bytes.into(), bytes.into());

        let control = zeroing_idxs(idxs).into();

        // Move index bit 4 into each byte's sign bit for VPBLENDVB.
        let select_high = x86::_mm256_slli_epi16::<3>(idxs.into());
        let from_low = avx2_half_pshufb(lolo, control);
        let from_high = avx2_half_pshufb(hihi, control);
        x86::_mm256_blendv_epi8(from_low, from_high, select_high).into()

@Shnatsel

Copy link
Copy Markdown
Member Author

You could, but according to llvm-mca that keeps the throughput unchanged and makes latency slightly worse, so there's no point in doing that.

@axnsan12

Copy link
Copy Markdown

Yeah I think my suggestion to use wrapping_add was wrong because I misremembered my experiments from ~1 year ago. saturating_add (expecting paddsb to be emitted) would actually be correct and not require the additional OR.

And IIRC after doing that the custom avx2 shuffle became unnecessary because LLVM just generated the same code using the generic recursive one

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants