x86 IL: Float -> int conversions are wrong for special values (e.g. +∞)

[moste00]

Work environment

Questions	Answers
OS/arch/bits (mandatory)	Ubuntu 24.04.3 LTS / x86 / 64 bits
File format of the file you reverse (mandatory)	N/A
Architecture/bits of the file (mandatory)	x86 / 64-bits
rizin -v full output, not truncated (mandatory)	rizin 0.9.0 @ linux-x86-64 commit: d220bca28a190f47ad8d61dfde97ea997155a868

The x86 spec, (found here) mandates that all special values (like +infinity, -infinity, and NANs) are handled differently when converted to integers via FIST and FISTP x87 instructions, as follows:

This can be confirmed by simply running the following code on any x86 machine.

#include <stdio.h>
#include <stdint.h>
#include <string.h>

int main(void) {
    uint32_t pinf_bits = 0x7F800000U; /* +infinity IEEE 754 single precision */
    float pinf_f;
    memcpy(&pinf_f, &pinf_bits, sizeof pinf_f);

    int32_t result = 0xDEADBEEF; /* sentinel: detects a no-write */

    __asm__ volatile (
        "flds   %1\n\t"   /* FLD: push +infinity as float32 onto x87 stack => ST(0) */
        "fistpl %0\n\t"   /* FISTP m32int: store ST(0) as int32 to memory, pop stack */
        : "=m"(result)
        : "m"(pinf_f)
        : "st"
    );

    printf("x86 FISTP(+inf) = 0x%08X  (%d)\n", (uint32_t)result, result);
    printf("Expected x86 integer-indefinite: 0x80000000\n");
    if ((uint32_t)result == 0x80000000U) {
        printf("hardware: PASS\n"); return 0;
    } else {
        printf("hardware: UNEXPECTED\n"); return 1;
    }
}

which prints

x86 FISTP(+inf) = 0x80000000  (-2147483648)
Expected x86 integer-indefinite: 0x80000000
hardware: PASS

However, this is not the current behaviour of the x86 lifter, which computes the value 0 instead for the conversion.

Expected behavior

The integer indefinite value as defined by the x86 specs is 0x800000..., it should be returned for all invalid conversion operation IF the invalid operation exception is masked. If the exception is not masked, then it should be raised and no value is stored in memory. (Although testing on the actual hardware without any special configuration or masking results in the integer indefinite value).

Actual behavior

The lifted instructions produces 0.

Steps to reproduce the behavior

Please see the attached files below. x86_hw_test_float_conversion.c is the code reproduced above which computes the correct value on hardware, and test_il_float_cast.c is a small script that uses the Rizin API to lift and run the same 2 instructions and reads their output. The Rizin test also prints the IL event stream.

Additional Logs, screenshots, source code, configuration dump, ...

The origins of this bug ultimately traces down as follows:

• In librz/arch/isa/x86/il_fp_ops.inc, the fistp lifter uses x86_il_int_from_floating to convert the top of the FP stack

• In librz/arch/isa/x86/common.c, x86_il_int_from_floating is a macro for a call of x86_il_int_from_floating_ctx

• Which ultimately runs F2SINT, which is call to rz_il_op_new_fcast_sint in ibrz/il/il_opcodes.c

• Evaled by rz_il_handler_fcast_sint in librz/il/theory_fbasic.c

• Converting the float value into an int by running rz_float_cast_sint in librz/util/float/float.c

As a side note, the rz_float_cast_int function in the same file also delegates unconditionally to rz_float_cast_sint, which might be another bug.

It's arguably rz_float_cast_sint that is the real root cause of the bug, as another function from the same file is rz_float_convert, which correctly guards against special values before manipulating the floating point value.

Other archiectures ?

This bug is very probably common and present in all archiectures that tries to use F2INT and F2SINT directly to convert floating point values, for example the ARM archiecture. The bug itself was first encountered during RISC-V testing of new lifter code, before being fixed via a workaround in the lifter itself.

It might be worth stopping before fixing the x86 symptoms and asking whether the cast_sint IL operations needs rethinking or a more configurable interface. Hardcoding the x86 behaviour into the IL is not desirable either, as other architectures needs to be able to use it without fighting against x86 bias.

x86_hw_test_float_conversion.c
test_il_float_cast.c

[moste00]

NOTE: +∞ was used as an example here, but any fix must be tested against -∞, NANs, etc...