Assembly Floating-Point — x87 FPU and SSE Math Operations

In this tutorial, you will learn about Assembly Floating. We cover key concepts, practical examples, and best practices to help you master this topic.

x86-64 floating-point arithmetic uses x87 FPU instructions for legacy math and SSE instructions for modern scalar float operations with better performance and fewer limitations.

What You'll Learn

• x87 FPU stack and instructions
• SSE scalar floating-point
• Converting between integer and float
• Calling conventions for float args

Why It Matters

Scientific computing, graphics, game physics, and financial calculations all need floating-point. Doda Browser uses SSE for CSS transform calculations in its rendering engine.

Real-World Use

3D graphics, physics simulation, audio processing, data visualization, and any application requiring real-number computation with fractional values.

flowchart LR
    A["Floating Point"] --> B["x87 FPU"]
    A --> C["SSE Scalar"]
    A --> D["SSE Packed"]
    B --> E["Stack-based"]
    C --> F["Register-based"]
    D --> G["SIMD vectors"]
    style A fill:#2563eb,stroke:#2563eb,color:#fff
    style B fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style C fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style D fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style E fill:#dbeafe,stroke:#2563eb,color:#1e40af

x87 FPU Stack

The x87 FPU uses an 8-element register stack (ST0 through ST7):

section .data
    val1 dq 3.14
    val2 dq 2.71

section .text
    ; Load values onto FPU stack
    fld qword [val1]        ; ST0 = 3.14
    fld qword [val2]        ; ST0 = 2.71, ST1 = 3.14

    ; Add top two stack elements
    faddp                   ; ST0 = ST0 + ST1, pop stack
    ; Now ST0 = 5.85

    ; Store result back to memory
    fstp qword [result]     ; store and pop

x87 Common Instructions

fld     ; load float to stack
fst     ; store float from stack
fstp    ; store and pop
fadd    ; add
faddp   ; add and pop
fsub    ; subtract
fsubp   ; subtract and pop
fmul    ; multiply
fmulp   ; multiply and pop
fdiv    ; divide
fdivp   ; divide and pop
fsqrt   ; square root
fchs    ; change sign
fabs    ; absolute value

SSE Scalar Float

Modern approach using XMM registers:

section .data
    a dd 3.14           ; single precision (float)
    b dd 2.71
    result dd 0.0

section .text
    ; Scalar single-precision
    movss xmm0, [a]     ; xmm0 = 3.14
    movss xmm1, [b]     ; xmm1 = 2.71
    addss xmm0, xmm1    ; xmm0 = 5.85
    movss [result], xmm0

    ; Scalar double-precision
    a_dq dq 3.14
    b_dq dq 2.71
    result_dq dq 0.0

    movsd xmm0, [a_dq]  ; xmm0 = 3.14
    movsd xmm1, [b_dq]  ; xmm1 = 2.71
    addsd xmm0, xmm1    ; xmm0 = 5.85
    movsd [result_dq], xmm0

SSE Arithmetic Instructions

addss   ; add scalar single
subss   ; subtract scalar single
mulss   ; multiply scalar single
divss   ; divide scalar single
sqrtss  ; square root scalar single

addsd   ; add scalar double
subsd   ; subtract scalar double
mulsd   ; multiply scalar double
divsd   ; divide scalar double
sqrtsd  ; square root scalar double

Integer-Float Conversion

section .data
    int_val dd 42
    float_val dd 0.0

section .text
    ; Integer to float
    cvtsi2ss xmm0, [int_val]    ; int -> float
    cvtsi2sd xmm0, [int_val]    ; int -> double

    ; Float to integer
    cvtss2si eax, xmm0          ; float -> int (truncate)
    cvttss2si eax, xmm0         ; float -> int (truncate, no rounding)

    ; With rounding
    cvtss2si eax, xmm0          ; round to nearest
    cvttss2si eax, xmm0         ; truncate toward zero

Square Root Example

section .data
    val dq 144.0
    result dq 0.0

section .text
    ; x87 style
    fld qword [val]
    fsqrt
    fstp qword [result]         ; result = 12.0

    ; SSE style (double)
    movsd xmm0, [val]
    sqrtsd xmm0, xmm0
    movsd [result], xmm0

    ; SSE style (float)
    val_f dd 144.0
    sqrtss xmm0, [val_f]

Function with Float Arguments

System V AMD64 passes float/double args in XMM0-XMM7:

; float sum_squares(float a, float b)
; a in XMM0, b in XMM1
sum_squares:
    mulss xmm0, xmm0        ; a * a
    mulss xmm1, xmm1        ; b * b
    addss xmm0, xmm1        ; a^2 + b^2
    ret

_start:
    movss xmm0, [val_a]
    movss xmm1, [val_b]
    call sum_squares
    ; result in xmm0

Common Mistakes

1. Mixing float and double sizes

Float is 32-bit, double is 64-bit. Using movss with a double or movsd with a float reads garbage.

2. x87 stack overflow

The FPU stack has only 8 entries. Pushing more than 8 values causes an exception.

3. Not popping x87 stack

Leaving values on the FPU stack causes overflow on subsequent FPU operations.

4. Using SSE without checking CPU support

All x86-64 CPUs support SSE2. No need to check for 64-bit code. 32-bit code should verify.

5. Comparing floats for equality

Use ucomiss/ucomisd with a tolerance. Exact equality rarely works due to rounding.

Practice Questions

1. What is the difference between x87 and SSE floating-point?

x87 uses a stack-based architecture with 80-bit internal precision. SSE uses register-based XMM registers with 32-bit or 64-bit precision.

2. How are float arguments passed in System V AMD64?

In XMM0-XMM7 (one register per argument).

3. What does cvtsi2ss do?

Converts a 32-bit integer (scalar) to a 32-bit float (scalar single).

4. Why should you avoid comparing floats with exact equality?

Floating-point rounding errors make exact comparison unreliable. Use a tolerance: |a - b| < epsilon.

Challenge: Write a program that computes the area of a circle given radius using SSE.

section .data
    radius dd 5.0
    pi dd 3.14159
    result dd 0.0

section .text
    global _start

_start:
    movss xmm0, [radius]
    mulss xmm0, xmm0           ; r^2
    mulss xmm0, [pi]           ; pi * r^2
    movss [result], xmm0

    ; result = 78.53975
    mov rax, 60
    xor rdi, rdi
    syscall

FAQ

{{< faq question="Is x87 deprecated?" >}} Intel recommends SSE/AVX for new code, but x87 is still supported. x87's 80-bit internal precision is still useful for certain numerical algorithms. {{< /faq >}}

{{< faq question="What precision do float and double have?" >}} Float has 23 bits of mantissa (~7 decimal digits). Double has 52 bits (~15 decimal digits). {{< /faq >}}

{{< faq question="How do I print floating-point values?" >}} Call printf with %f format string. Pass the value in XMM0 and set EAX=1 (number of XMM registers used). {{< /faq >}}

{{< faq question="What is NaN and how is it handled?" >}} NaN (Not a Number) results from invalid operations like 0/0. SSE comparisons with NaN set the parity flag. Use jp to detect NaN. {{< /faq >}}

{{< faq question="Can I use SIMD for scalar operations?" >}} Yes, SSE scalar instructions use the same XMM registers but only operate on the lowest element. The upper elements are preserved. {{< /faq >}}

Mini Project

Write a program that computes the quadratic formula for a given a, b, c.

section .data
    a dq 1.0
    b dq 5.0
    c dq 6.0
    four dq 4.0
    two dq 2.0
    root1 dq 0.0
    root2 dq 0.0

section .text
    global _start

_start:
    movsd xmm0, [b]         ; b
    movsd xmm1, [b]
    mulsd xmm1, xmm1        ; b^2
    movsd xmm2, [four]
    movsd xmm3, [a]
    movsd xmm4, [c]
    mulsd xmm2, xmm3        ; 4*a
    mulsd xmm2, xmm4        ; 4*a*c
    subsd xmm1, xmm2        ; discriminant = b^2 - 4ac
    sqrtsd xmm5, xmm1       ; sqrt(discriminant)

    movsd xmm6, [b]
    subsd xmm6, zero         ; -b
    subsd xmm0, xmm5         ; -b - sqrt(d)
    movsd xmm1, [two]
    mulsd xmm1, [a]          ; 2a
    divsd xmm0, xmm1
    movsd [root1], xmm0

    mov rax, 60
    xor rdi, rdi
    syscall

section .data
    zero dq 0.0

What's Next

Now that you understand floating-point, proceed to SSE SIMD instructions.