Assembly Procedures — Function Calls and Calling Conventions

In this tutorial, you will learn about Assembly Procedures. We cover key concepts, practical examples, and best practices to help you master this topic.

Assembly procedures use call/ret instructions with the stack to implement function calls following the System V AMD64 ABI calling convention for parameter passing and register preservation.

What You'll Learn

• The call stack and stack frame
• The System V AMD64 calling convention
• Caller vs callee-saved registers
• Passing arguments and return values

Why It Matters

Procedures are fundamental to all structured programming. Understanding the calling convention is essential for debugging, reverse engineering, and writing assembly that interfaces with C libraries.

Real-World Use

Function calls in compiled code, C library interop, operating system kernel calls, and reverse engineering malware to understand its subroutine structure.

flowchart LR
    A["Caller"] --> B["call func"]
    B --> C["Stack Frame"]
    C --> D["Function Body"]
    D --> E["ret"]
    E --> F["Caller Resumes"]
    G["Arguments"] --> C
    H["Return Value"] --> F
    style A fill:#2563eb,stroke:#2563eb,color:#fff
    style B fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style C fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style D fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style E fill:#dbeafe,stroke:#2563eb,color:#1e40af

Basic Procedure

section .text
    global _start

_start:
    call my_proc        ; push return address, jump to my_proc
    mov rax, 60         ; exit
    xor rdi, rdi
    syscall

my_proc:
    ; procedure body
    mov rax, 42
    ret                 ; pop return address, jump back

Arguments in Registers

System V AMD64 calling convention:

; add_two(a, b) -> a + b
add_two:
    mov rax, rdi        ; rax = first arg
    add rax, rsi        ; rax += second arg
    ret

_start:
    mov rdi, 10         ; first arg
    mov rsi, 20         ; second arg
    call add_two
    ; rax = 30

Caller vs Callee-Saved Registers

; Callee-saved: RBX, RBP, R12-R15
;   Must preserve these values
; Caller-saved: RAX, RCX, RDX, RSI, RDI, R8-R11
;   Can freely use (caller saves if needed)

my_proc:
    push rbx            ; save rbx
    push rbp            ; save rbp

    mov rbx, rdi        ; use rbx
    mov rbp, rsi        ; use rbp
    ; ... procedure body ...

    pop rbp             ; restore rbp
    pop rbx             ; restore rbx
    ret

Stack Frame

my_function:
    ; Prologue
    push rbp            ; save old base pointer
    mov rbp, rsp        ; set new base pointer

    ; Allocate local variables
    sub rsp, 32         ; reserve 32 bytes for locals

    ; Body
    mov [rbp - 8], rdi  ; local var at rbp-8
    mov [rbp - 16], rsi ; local var at rbp-16

    ; Epilogue
    mov rsp, rbp        ; restore stack pointer
    pop rbp             ; restore base pointer
    ret

Accessing Stack Parameters

; For functions with more than 6 arguments
; Arguments 7+ are passed on the stack

func_with_many_args:
    push rbp
    mov rbp, rsp

    ; arg1 = rdi
    ; arg2 = rsi
    ; arg7 = [rbp + 16]   (above return address and saved rbp)
    ; arg8 = [rbp + 24]

    mov rax, [rbp + 16]  ; 7th argument

    pop rbp
    ret

Calling C Library Functions

extern printf
extern exit

section .data
    fmt db "Result: %d", 10, 0

section .text
    global _start

_start:
    mov rdi, 42          ; first arg to square
    call square
    mov rsi, rax         ; result as second arg to printf
    mov rdi, fmt         ; format string
    call printf

    mov rdi, 0
    call exit

square:
    mov rax, rdi
    imul rax, rax
    ret

Assemble with: nasm -f elf64 file.asm && gcc -no-pie file.o -o file

Recursive Functions

; factorial(n) -> n!
factorial:
    cmp rdi, 1
    jle base_case

    push rdi             ; save n
    dec rdi
    call factorial        ; factorial(n-1)
    pop rdi              ; restore n
    imul rax, rdi        ; n * factorial(n-1)
    ret

base_case:
    mov rax, 1
    ret

_start:
    mov rdi, 5
    call factorial
    ; rax = 120

Common Mistakes

1. Mismatched push/pop

Every push needs a matching pop. Unbalanced stack causes ret to jump to the wrong address.

2. Forgetting to preserve callee-saved registers

If a function modifies RBX without saving it, the caller crashes when using its own RBX value.

3. Stack alignment

The System V ABI requires 16-byte stack alignment at the point of a call. Violations cause crashes in SSE/AVX instructions.

4. Not zeroing RBP in leaf functions

If a leaf function doesn't touch the stack, it can skip the prologue but must not use RBP-based addressing.

5. Using 32-bit registers for return values

int functions return in EAX (32-bit). long functions return in RAX (64-bit). The upper 32 bits of RAX are undefined for 32-bit returns.

Practice Questions

1. What registers hold the first 6 integer arguments in System V AMD64?

RDI, RSI, RDX, RCX, R8, R9.

2. What is the purpose of the prologue push rbp; mov rbp, rsp?

It saves the caller's base pointer and sets up a fixed reference point for local variables and parameters.

3. Why must callee-saved registers be preserved?

The caller expects them to have the same value after the call returns. Violations cause subtle bugs.

4. What is the stack alignment requirement for calls?

The stack must be 16-byte aligned before the call instruction, meaning RSP mod 16 == 0 at the function entry point.

Challenge: Write a recursive assembly function that computes the nth Fibonacci number.

fibonacci:
    cmp rdi, 0
    je return_0
    cmp rdi, 1
    je return_1

    push rdi
    dec rdi
    call fibonacci        ; fib(n-1)
    pop rdi
    push rax              ; save fib(n-1)
    sub rdi, 2
    call fibonacci        ; fib(n-2)
    pop rdi               ; restore fib(n-1)
    add rax, rdi          ; fib(n-1) + fib(n-2)
    ret

return_0:
    xor rax, rax
    ret

return_1:
    mov rax, 1
    ret

FAQ

{{< faq question="What happens if I don't follow the calling convention?" >}} The code still runs but may crash when calling external libraries or when the caller optimizes based on ABI guarantees. {{< /faq >}}

{{< faq question="Can I use any register for arguments?" >}} Yes, but if you call C functions, you must follow the System V ABI. For internal procedures, you can use any convention you define. {{< /faq >}}

{{< faq question="How many arguments can be passed in registers?" >}} Six integer arguments in registers. Additional arguments go on the stack. {{< /faq >}}

{{< faq question="What is a leaf function?" >}} A function that calls no other functions. Leaf functions can omit the prologue/epilogue for efficiency. {{< /faq >}}

{{< faq question="How do I return a struct from a function?" >}} The caller passes a hidden pointer argument in RDI for the return struct. The function writes to that pointer and returns it in RAX. {{< /faq >}}

Mini Project

Write a program that uses procedures to compute the sum of squares from 1 to n.

section .data
    result_msg db "Sum of squares: ", 0

section .text
    global _start

_start:
    mov rdi, 10
    call sum_of_squares
    ; rax now has sum of 1^2 + 2^2 + ... + 10^2

    mov rdi, rax
    mov rax, 60
    syscall

sum_of_squares:
    xor rax, rax
    xor rcx, rcx

.loop:
    inc rcx
    mov rdx, rcx
    imul rdx, rdx
    add rax, rdx
    cmp rcx, rdi
    jl .loop
    ret

What's Next

Now that you understand procedures, proceed to system calls.