Skip to content

erts: Fix float16 construction in the software fallback#11334

Open
a1x-an wants to merge 1 commit into
erlang:maintfrom
a1x-an:erts/fix-fp16-fallback-rounding
Open

erts: Fix float16 construction in the software fallback#11334
a1x-an wants to merge 1 commit into
erlang:maintfrom
a1x-an:erts/fix-fp16-fallback-rounding

Conversation

@a1x-an

@a1x-an a1x-an commented Jul 3, 2026

Copy link
Copy Markdown

Fixes GH-11332.

The software float16 conversion used when _Float16 is not available
(notably all official Windows/MSVC builds) produced wrong bit patterns for
nearly all values that are subnormal in binary16, and misrounded some normal
values, so the emulator could not re-construct a value it had just matched:

1> <<F:16/float>> = <<0,51>>.   %% 51 × 2⁻²⁴, exactly representable
2> <<F:16/float>>.
<<0,39>>                        %% construct(match(B)) ≠ B

Three defects in f32_to_f16() (erts/emulator/beam/erl_bits.c):

  1. The subnormal branch OR:ed the implicit leading bit into position 23
    after the mantissa had already been reduced to its top 10 bits, then
    shifted by an exponent-derived amount - producing garbage for all
    binary16 subnormals except a few accidental fixed points. The rounding
    increment was also a full ULP (1 << shift) where half an ULP is
    required.
  2. The normal branch ignored bits 9..0 in its sticky test, so values just
    above a rounding midpoint whose sticky bits lie only in the low bits
    rounded down as if they were ties (when the kept LSB was even).
  3. Converting through float (f32_to_f16((float) x)) rounds twice
    (double→float→binary16), so results could differ from the native
    single-rounding (_Float16) x path even with 1) and 2) fixed:
    <<1.000488282181322574615478515625:16/float>> (= 1 + 2⁻¹¹ + 2⁻³⁰)
    gave 0x3C00 through the fallback but 0x3C01 on native builds.

The fallback is replaced by a direct integer-exact double→binary16
conversion with a single round-to-nearest-even step, giving results
identical to the native path. The match direction (f16_to_f32) was
correct and is unchanged.

Verification

  • Standalone harness against a compiler-provided _Float16 oracle
    (gcc 13): exhaustive sweep of all finite float32 inputs - 0 mismatches for
    the new conversion (the current one disagrees on 417,333,236 inputs);
    construct∘match identity over all finite binary16 values - 0 failures
    (currently 2026); 10⁹ random finite doubles - 0 mismatches.
  • Cross-compiler check: the same source compiled with MSVC (VS 2022,
    cl /O2 - the toolchain of the official Windows builds, which lacks
    _Float16 and therefore uses this fallback) and with gcc -O2
    produces bit-identical outputs for both the old and the new conversion
    over all finite float32 inputs (FNV-1a digests over the full sweep
    match exactly), so the fix is compiler-independent.
  • Patched emulator built from maint on x86_64 Linux (musl/gcc):
    • native _Float16 path: all witness values and the full binary16
      identity sweep pass (unchanged behaviour);
    • fallback path forced by undefining FLOAT16_IS_CONVERTIBLE: all
      witness values and the full identity sweep pass, including the
      double-rounding witness - the fallback now matches the native path
      exactly.
  • The shipped Windows 29.0.3 emulator (which uses the fallback)
    reproduces all failures; the new bs_construct_SUITE:fp16/1 cases fail
    on it and pass with the fix.

New test cases cover exact subnormals (previously corrupted), a
sticky-bits rounding witness, an exact tie (round-to-even), the
double-rounding witness, and a construct∘match identity sweep over every
finite binary16 value.

@CLAassistant

CLAassistant commented Jul 3, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

CT Test Results

    3 files    136 suites   50m 49s ⏱️
1 682 tests 1 626 ✅ 56 💤 0 ❌
2 325 runs  2 251 ✅ 74 💤 0 ❌

Results for commit 6530318.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

// Erlang/OTP Github Action Bot

@jhogberg jhogberg self-assigned this Jul 5, 2026
@jhogberg jhogberg added testing currently being tested, tag is used by OTP internal CI team:VM Assigned to OTP team VM labels Jul 5, 2026
@jhogberg

jhogberg commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Thanks for the PR! Can you rebase this onto the tag patch-base-28 so that we can release a patch on OTP 28? :-)

The software conversion used when _Float16 is not available (notably
all official Windows/MSVC builds) produced wrong bit patterns for
nearly all values that are subnormal in binary16, and misrounded some
normal values:

    1> <<F:16/float>> = <<0,51>>.        %% 51 * 2^-24, exact
    2> <<F:16/float>>.
    <<0,39>>                             %% construct(match(B)) =/= B

The subnormal branch of f32_to_f16() OR:ed the implicit leading bit
into position 23 after the mantissa had already been reduced to its
top 10 bits, and used a whole ULP as the rounding increment. The
normal branch ignored bits 9..0 in its sticky test, so values just
above a rounding midpoint could round down as if they were ties.
Additionally, converting through float (double -> float -> binary16)
rounds twice, so results could differ from the native _Float16 path
for the same input.

Replace the fallback with a direct integer-exact double -> binary16
conversion with a single round-to-nearest-even step, giving the same
results as the native path. The match direction (f16_to_f32) was
correct and is unchanged.

Verified exhaustively against a compiler-provided _Float16 oracle:
all 2^32 float32 inputs, construct/match identity over all finite
binary16 values, and 10^9 random doubles produce identical results.
Also verified in a patched emulator built from maint on x86_64 Linux
with the fallback both disabled (native path, unchanged) and forced
via FLOAT16_IS_CONVERTIBLE (fixed path, now identical to native).
@a1x-an a1x-an force-pushed the erts/fix-fp16-fallback-rounding branch from 9aa1b6f to 6530318 Compare July 9, 2026 12:45
@a1x-an

a1x-an commented Jul 9, 2026

Copy link
Copy Markdown
Author

Done, rebased onto patch-base-28 :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

team:VM Assigned to OTP team VM testing currently being tested, tag is used by OTP internal CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants