Itoa implementations for NEON, SSE2, SSE4.1, AVX2 - #156
TobiSchluter wants to merge 5 commits into
Conversation
Self-contained in zmij-int.cc under namespace details_int so its helpers cannot collide with the floating-point implementation's, including in unity builds; the public write overloads and buffer sizes live in the shared zmij.h. Kernels by ISA tier: NEON (including a fused signed-i32 body), SSE4.1, AVX2 256-bit u128 chunk passes, plain SSE2 and a scalar SWAR fallback. The Zen 5-tuned u64 splits (the float-reciprocal 4+16 peel and the 12+8 XMM-tail split) sit behind tune-gated macros: both offload work to vector ports that only helps where those are split from the scalar ports; on Tiger Lake the 12+8 split measured +22-31% against the generic 16+4 peel. A single up-front dynamic-object-size bounds check per conversion replaces the fortified per-copy checks, which cost a call plus clamp branches once the kernels are inlined into a frame with a visible destination buffer.
zmij-itoa-test.cc compiles into every flag-variant test build and includes zmij-int.cc directly, like zmij-impl-test.cc, so each variant tests its own configuration. int-check exhaustively verifies all 32-bit values against fmt; itoa-benchmark compares against fmt across log-uniform sweeps and realistic digit-count streams.
Scale the value by 10^(16 - len) from a table so the 16-digit kernel's output is already left-aligned and stores straight from the XMM register, replacing the GPR extraction and 128-bit right shift. The 33-39-digit u128 top group goes through a new itoa_head7 (one to_bcd8 + 8-byte store) instead of the 16-digit kernel, whose scaled form would wait on the digit count at the end of the two-divmod chain. On a Ryzen 9 9950X this makes the SSE2 tier faster by means of 3-6% (gcc 16) and 8-13% (clang 21) on the u64/i64 and u128 streams, with the 33-39-digit band at or below the old times. A fused variant folding the scale into the /1e8 split via per-row reciprocals measured slower everywhere and was dropped.
|
I forgot to say two things:
|
vitaut
left a comment
vitaut
left a comment
There was a problem hiding this comment.
Impressive work!
Overall looks good but please address inline comments. Also I would drop the exhaustive test. It is useful for float because of hard-to-test boundary conditions but I don't think it's particularly useful for integers.
| __m128 shifted = _mm_castsi128_ps(_mm_slli_si128(_mm_castps_si128(qf), 4)); | ||
| // digit_k + '0' + 2^23 = (qf_k + bias) - 10 * qf_{k-1} | ||
| __m128i dig = | ||
| _mm_castps_si128(_mm_fnmadd_ps(ten, shifted, _mm_add_ps(qf, bias))); |
There was a problem hiding this comment.
This need to check whether FMA is enabled.
There was a problem hiding this comment.
This block is conditional on ZMIJ_USE_AVX2_U64_FP which in turn checks for ZMIJ_USE_AVX2 && (defined(__znver5__) || defined(__tune_znver5__)).
Other than that, I just learned from StackOverflow that there is a CPU out in the wild that has AVX2 but not FMA, which seems like a mistake. So, I don't think a local check is mandated, but there might be a CPU with FMA and no AVX2 (and lots of hardware multipliers) that could benefit from this?
There was a problem hiding this comment.
Oh, I think I get it now: if the user selects an -march that enables AVX2 but not FMA they might see an error if they also enable ZMIJ_USE_AVX2_U64_FP. Gotta love the randomness of the x86 instruction set growth! Ok, I will make sure that the checks are tight.
More as a note to self, this also applies to itoa_top8 below (which is I think the most creative thing I did in all of this, and which I failed to highlight in the PR).
There was a problem hiding this comment.
if the user selects an -march that enables AVX2 but not FMA they might see an error
Yes, that was my concern.
There was a problem hiding this comment.
I tried to address this concern with some infrastructure in #160
| // standalone with the same settings; identical redefinitions keep unity | ||
| // builds that include both files valid. | ||
|
|
||
| #ifndef ZMIJ_USE_SIMD |
There was a problem hiding this comment.
I think we should put common configuration macros in zmij.h to avoid duplication.
|
Would you be able to also make a C version of this? MoarVM uses the C floating point code, and I would love to use Żmij for integers as well. |
|
Since my Zen5 broke, I moved to an i9 (Raptor Lake). Great to physically experience a modern Intel CPU that doesn't have AVX512. Performance is distributed a bit differently -- sign handling seems to cost more. I added the benchmark table to the first post. One curious observation that I made because I inadvertently ran the floating-point benchmarks as well is that for |
Intermediate calculation overflowed, leading to overweighting of largest bin.
itoa buffers are one larger than utoa buffers, missing one byte in the buffer overrun check.
3 participants
Hi Victor,
I've been watching your progress, and it's been impressive. What I did, isn't so impressive and it took me a while to get around to this, but I'm finally proposing a pull request for the
itoastuff. Headlines first: branch-free (except 128bits) implementations of integer itoa (signed and unsigned <= 32bit, 64bit, and 128bit where available). Predictable runtime down to under 1ns on M5 neon. Winner on almost all benchmarks I could think of.In order to avoid crashing with your changes, and making the review a bit easier by not interleaving the two, I've split the integer handling into a separate file. Mainly because I reuse
to_bcd8which counts the trailing zero some unnecessary stuff is present, but I think the unrelated stuff is easier to ignore this way.What is this optimized for? The idea I had was that the tpyical, performance-aware user will convert one number, and then either copy the string somewhere else or continue inserting characters after each converted integer. Notably, they won't actually look at the emitted bytes directly, and only care about the length of the emitted string which the CPU's out-of-order implementation will fill in while the program continues. So the first thing to optimize is the length evaluation. Here I used the algorithm found in fmt with three modifications:
clzbut on old intel CPUs this is implemented asbtzfollowed byxor 63. REversing the table order makes thexorunnecessary. Depending on the compile target the optimal sequence is chosen. The tables themselves are evaluated at compile time.As for the conversions themselves, it is perhaps worth pointing out that a decimal number string always takes more than twice the space than the binary number. An 32bit number takes up to ten bytes, a 64bit number up to 20, a 128bit number up to 39. Obviously, these numbers don't map well to the registers which are 8, 16 or (AVX2) 32 bytes wide. Therefore, the numbers need to be split, and because SIMD doesn't have full-width 64bit multiplies, they actually need to be split several times. Most of the work went into identifying the optimal splitting sequences. Add to that that on Intel there are several generations of SIMD instructions, that perform differently across different machines, and you will understand that it took some time to come up with what I think are the best sequences.
AI summary of the digit-group splits in the zmij integer conversion
Every SIMD tier shares the same 16-digit kernel underneath: the GPR side does
v / 1e8to get two 8-digit halves into two 64-bit lanes, then the vector sidesplits each lane 8 → 4+4 (
mul_epu32by the 1e4 reciprocal), 4 → 2+2(
mulhi_epu16by the 100 reciprocal), 2 → 1+1 (mulhiby the 10 reciprocal),and a final shuffle reverses and trims leading zeros. The differences below are
about what is fed into that kernel and what is handled outside it.
u32 (up to 10 digits)
v/100andv/1e6in parallel, feedingto_bcd8_split(SWAR 3-step on scalar, SSE2to_bcd_4x4on SSE2), right-shifted to trim and stored as 8 bytes; low 2 digits via thedigits2tablev/100andv/1e6; the 8-digit part is packed to two 4-digit lanes with one madd, the 2-digit remainder goes in a third lane. Oneto_ascii_4x4pass and one pshufb fromrevalign_shuffle10, one 16-byte storev/1e8(0..42) packed with the remainder by one madd, full 16-digit body, one store. Signed i32 folds the-into the same store via a widened shuffle window and asign_biasrowu64 (up to 20 digits)
v/1e8andv/1e16(parallel). Top ≤ 4 digits via divmod100 + twodigits2lookups, mid and low viato_bcd8SWAR. Assembled right-aligned in a 48-byte buffer, one 20-byte copy at 24 − lenv/1e4in GPR; body = high if v ≥ 1e16 else v, so body < 1e16. Body pre-scaled by 10^(16−len) so the kernel output is already left-aligned, no shuffle. Low 4 digits always written after via twodigits2lookups, counted only if v ≥ 1e16revalign_shufflebody instead of the scale trick; low 4 digits viadigits2ZMIJ_USE_U64_SPLIT12(Zen 5 default)v/1e16andv/1e8as two independent reciprocal multiplies, no chaining. Lanes (q16, q8 mod 1e8) go through the 16-digit kernel with an unclamped shuffle offset of 24 − c, which emits all-padding for v < 1e8. Low 8 digits viasplit10k+to_ascii_4x4+ arevalign_shufflewindow, 8-byte store at out + klenZMIJ_USE_AVX2_U64_FP(Zen 5 default)to_ascii4_pson the FP ports, trimmed by thepeel4_packwindow, 4-byte store. Remainder lanes throughitoa_body_lanesat out + hlen, whose 16-byte store overwrites the head's garbagev/1e4via umulh with astatic_datareciprocal, compare against 1e16 − 1 from the same ldp. Low 4 via a /100 reciprocal and twodigits2lookupsu128 (only above
UINT64_MAX, else the u64 path)First peel: a 128-bit reciprocal divmod by 1e16 gives the low 16 digits and a
quotient. If the quotient is below 1e16 (19 to 32 digits total) the split is
top ≤ 16 trimmed + low 16 padded. Otherwise a second narrow divmod (a 64-bit
reciprocal for 5^16 on quotient >> 16) gives top ≤ 7 + mid 16 + low 16.
to_bcd8, one 32-byte copy from a 64-byte bufferto_bcd8, one 40-byte copyto_ascii16via pshufd, no bswap)itoa_head7: one 8-digit BCD group, shifted and stored as 8 bytes, then 16 + 16 paddeditoa_top8: four base-100 blocks via float reciprocals, one SWAR /10 split, one pshufb trim, 8-byte store; then 32 padded in one 256-bit passrevalign_shuffleis the pure reversal)itoa_body32_padSigned wrappers
i128 first checks whether the value fits in i64 and takes that path. For
everything else the sign is written first and the unsigned path runs, except
i32 on NEON, which has the dedicated single-store body above.
Note on AVX2
AVX2 without the Zen 5 tuning flag does not get its own u64 split. The first
branch in
itoarequires eitherZMIJ_USE_AVX2_U64_FPorZMIJ_USE_U64_SPLIT12, so a plain-mavx2build falls to the same 16 + 4 peelas generic SSE4.1, and only the u128 paths differ.
As for the floating point conversions, AVX2 turned out to be quite useless. I managed to find two uses though: for large u128, the low 32bit digits can be handled in one AVX2 chain. Additionally, on architectures with lots of wide multipliers (Zen5) it is possible to use the floating point fma instruction to convert 4 digits in a minimum of operations. This is used for the 4 remaining digits after a 20 digits 64-bit number is split in to 16+4 digits.
As for benchmarks, in my fork of the itoa-benchmark repo you can find a number of low-level benchmarks. The current plots are for the same code as the one in this PR. For the PR, I chose to define a smaller set of benchmarks, and I looked for some more inspiration. Champagne and Lemire used a "Twitter JSON" dataset to benchmark their AVX512 itoa code. Unfortunately, that sample only contains some 2000 numbers -- an amount the branch prediction in a modern CPU easily trivializes, so I added a Markov-chain generator that is trained on that sample but generates much longer samples. This is the twitter benchmark, each number is converted,
", "is inserted after the number and the next number is converted. For this benchmark, I added a variety that always uses u64 conversion and one that picks the conversion based on the size of the number (most are smaller than <2e32, and so it is fairly predictable). The labels are zmij and zmij32 , respectively.Another benchmark converts i8 and u8 numbers, either as 32bit or as 64bit integers. This is a bit of a worst-case, as it can be handled trivially via a table, but still the 32bit code wins against fmt (the baseline for comparison). This also the only loss against fmt in the benchmarks.
Finally, and maybe not really applicable to the real world, I measured log-uniform distributions. Here we routinely see speed increases which reach more than an order of magnitude over fmt.
Detailed benchmark results
Note that the table show a few variants not included in the PR for the twitter benchmark: "iid" is a flat generator which picks from the set of twitter.json numbers. mk1 is a Markov-chain generator, i.e. the next number is picked on the conditional distribution given by the length of the current number. The mk2 is the default included in the repo, and it makes the number's length depend on the two preceding numbers.
All values are average nanoseconds per conversion. Our columns also contain the relative speed up compared to fmt.
AMD Ryzen 9 9950X (Zen 5), gcc 16
AMD Ryzen 9 9950X (Zen 5), clang 21
Apple M5, Apple clang 21
Apple M1 (192.168.1.98), Apple clang 21
Intel Core i7-1165G7 (Tiger Lake), gcc 16
Intel Core i7-1165G7 (Tiger Lake), clang 21
13th Gen Intel Core i9-13900KF (Raptor Lake), gcc 16
13th Gen Intel Core i9-13900KF (Raptor Lake), clang 21
I mentioned Champagne and Lemire and their AVX512 implementation above. I have not yet included AVX512 code for two reasons: 1. it is a wide field, there are so many variations of instructions available and different CPUs implement them with different characteristics, so it becomes hard to settle on one or two variants. Also, I mentioned above that CL use branches, and their data sets are not actually large enough to beat the branch predictor, so I would also have to demonstrate that I didn't tune my benchmarks just to beat them. All this seems to be too much work, especially given that distributions still don't push their defaults past v3 microarchitectures, and as I learned a few days ago, there really are users that are bound to v1 for years to come and who are dear to my heart given that I did my PhD at CERN. This actually triggered the last commit which gained almost 25% on the SSE2 path.
One aside: I didn't use fmt's
format_intbecause it has one defect that killed its usability: the output buffer has no padding. So It is not possible to copy the result out with a copy operation of predetermined length but instead a variable-lengthmemcpyis needed, which introduces a second hard-to-predict branch on top of the loop over digits.I should mention that the ARM {u,i}32 implementation is a bit over-optimized: M5 decodes 8 instructions at a time, so making the code aligned and < 32 instructions allows a benchmark to process it in 4 cycles. For this reason, and only this reason there is a variant of the zeros array that includes the leading '-' (that allowed shaving off one instruction at the expense of 16bytes more table data, and the re-arranging of the static_data: ARM can encode small offsets efficiently, so it is carefully layed out to make the relevant constants (mostly for the digit_count) available near the beginning of the struct with some dirty offset tricks. Given how short this function is, it may be a better choice to actually make it inlinable. This was also the main driver to place this code in a separate file, because you had also rearranged
static_datato use more efficient addressing on ARM. One could put the things integers need inside a nested struct and then get ideal addressing for both floats and integers, but it is micro-optimization churn that obscures what is actually going on, and also a continuous source of merge conflicts, whereas in this way AI can handle the folding back of the code easily.As for how to merge it, as I said, I moved it to a separate file to be able to easier review it, but I see not fundamental reason to not merge it back into the original file.