Calling Conventions and ABI — Function Calls in Compilers

In this tutorial, you'll learn about Calling Conventions and ABI. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Calling conventions define the low-level protocol for function calls — how arguments are passed, how return values are delivered, which registers must be preserved, and how the stack is managed — forming the Application Binary Interface (ABI) contract between caller and callee.

What You'll Learn & Why It Matters

In this tutorial, you will learn how calling conventions work, the major conventions (System V AMD64, ARM AAPCS, Microsoft x64), how to implement function prologues and epilogues, and how to design custom conventions. Understanding calling conventions is essential for writing compiler backends, implementing foreign function interfaces, and debugging at the assembly level.

Real-world use: The Rust compiler implements the Rust ABI with custom calling conventions that enable its ownership model, while also supporting C FFI through the System V AMD64 ABI. Durga Antivirus Pro uses calling convention knowledge to reconstruct function boundaries and arguments from stripped executables.

Prerequisites

You should understand code generation basics and register allocation. Familiarity with C programming and assembly language is assumed.

The Call Stack and Stack Frame

When a function is called, a new stack frame is created to hold local variables, saved registers, and return address.

graph TD
    subgraph "Stack Frame Layout (x86-64)"
        A["High Addresses"]
        B["Caller's Frame"]
        C["Return Address"]
        D["Saved RBP"]
        E["Local Variables"]
        F["Callee-Saved Regs"]
        G["Argument Build Area"]
        H["Low Addresses (RSP)"]
    end
    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    style C fill:#FF9800,color:#fff
    style D fill:#2196F3,color:#fff
    style E fill:#4CAF50,color:#fff

System V AMD64 Calling Convention

The most common convention on Linux and Unix systems uses register-first argument passing.

class SystemVABI:
    def __init__(self):
        self.int_arg_regs = ['rdi', 'rsi', 'rdx', 'rcx', 'r8', 'r9']
        self.float_arg_regs = ['xmm0', 'xmm1', 'xmm2', 'xmm3',
                               'xmm4', 'xmm5', 'xmm6', 'xmm7']
        self.return_reg_int = 'rax'
        self.return_reg_float = 'xmm0'
        self.callee_saved = ['rbx', 'rbp', 'r12', 'r13', 'r14', 'r15']

    def generate_call(self, func_name, arg_types):
        int_reg_idx = 0
        float_reg_idx = 0
        stack_args = []
        instructions = []

        for arg_type in arg_types:
            if arg_type == 'int' and int_reg_idx < len(self.int_arg_regs):
                reg = self.int_arg_regs[int_reg_idx]
                instructions.append(f"  mov {reg}, arg{int_reg_idx}")
                int_reg_idx += 1
            elif arg_type == 'float' and float_reg_idx < len(self.float_arg_regs):
                reg = self.float_arg_regs[float_reg_idx]
                instructions.append(f"  movsd {reg}, farg{float_reg_idx}")
                float_reg_idx += 1
            else:
                stack_args.append(arg_type)

        instructions.append(f"  call {func_name}")
        if stack_args:
            instructions.append(f"  add rsp, {len(stack_args) * 8}")
        instructions.append("  ; return value in rax/xmm0")
        return instructions

    def generate_function_prologue(self, local_var_size=0):
        instrs = ["  push rbp", "  mov rbp, rsp"]
        if local_var_size > 0:
            instrs.append(f"  sub rsp, {local_var_size}")
        return instrs

    def generate_function_epilogue(self):
        return ["  mov rsp, rbp", "  pop rbp", "  ret"]

abi = SystemVABI()
call = abi.generate_call('foo', ['int', 'int', 'int', 'float', 'int'])
for instr in call:
    print(instr)

Expected output:

  mov rdi, arg0
  mov rsi, arg1
  mov rdx, arg2
  movsd xmm0, farg3
  mov rcx, arg4
  call foo
  ; return value in rax/xmm0

ARM AAPCS Calling Convention

The ARM Architecture Procedure Call Standard uses four registers for integer arguments.

class AAPCS:
    def __init__(self):
        self.int_arg_regs = ['r0', 'r1', 'r2', 'r3']
        self.callee_saved = ['r4', 'r5', 'r6', 'r7', 'r8', 'r9', 'r10', 'r11']
        self.return_reg = 'r0'

    def generate_prologue(self):
        return ["  push {fp, lr}", "  mov fp, sp",
                "  ; sp must remain 8-byte aligned"]

    def generate_call(self, func_name, num_int_args):
        instructions = []
        for i in range(min(num_int_args, 4)):
            instructions.append(f"  mov r{i}, arg{i}")
        if num_int_args > 4:
            for i in range(4, num_int_args):
                instructions.append(f"  ldr r4, =arg{i}")
                offset = (i - 4) * 4
                instructions.append(f"  str r4, [sp, #{offset}]")
        instructions.append(f"  bl {func_name}")
        return instructions

    def generate_epilogue(self):
        return ["  mov sp, fp", "  pop {fp, lr}", "  bx lr"]

arm_abi = AAPCS()
call = arm_abi.generate_call('bar', 5)
for instr in call:
    print(instr)

Expected output:

  mov r0, arg0
  mov r1, arg1
  mov r2, arg2
  mov r3, arg3
  ldr r4, =arg4
  str r4, [sp, #0]
  bl bar

Callee-Saved vs Caller-Saved Registers

This distinction determines which registers must be preserved across function calls.

class RegisterSaveManager:
    def __init__(self, convention='systemv'):
        mapping = {
            'systemv': ['rbx', 'rbp', 'r12', 'r13', 'r14', 'r15'],
            'aapcs': ['r4', 'r5', 'r6', 'r7', 'r8', 'r9', 'r10', 'r11'],
        }
        self.callee_saved = mapping[convention]
        self.live_registers = set()

    def mark_live(self, reg):
        self.live_registers.add(reg)

    def generate_save_code(self):
        to_save = [r for r in self.callee_saved if r in self.live_registers]
        if not to_save:
            return []
        return [f"  push {', '.join(to_save)}"]

    def generate_restore_code(self):
        to_save = [r for r in self.callee_saved if r in self.live_registers]
        if not to_save:
            return []
        to_save.reverse()
        return [f"  pop {', '.join(to_save)}"]

    def should_spill_before_call(self, reg):
        caller_saved = {
            'systemv': ['rax', 'rcx', 'rdx', 'rsi', 'rdi', 'r8', 'r9', 'r10', 'r11'],
            'aapcs': ['r0', 'r1', 'r2', 'r3'],
        }
        return reg in caller_saved

manager = RegisterSaveManager('systemv')
manager.mark_live('rbx')
manager.mark_live('r12')
manager.mark_live('r13')

for instr in manager.generate_save_code():
    print(instr)
for reg in ['rbx', 'rcx', 'rdx', 'r12']:
    status = "caller-saved" if manager.should_spill_before_call(reg) else "callee-saved"
    print(f"  {reg}: {status}")

Expected output:

  push rbx, r12, r13
  rbx: callee-saved
  rcx: caller-saved
  rdx: caller-saved
  r12: callee-saved

Common Errors in Calling Convention Implementation

Error 1: Stack Misalignment

The System V AMD64 ABI requires RSP to be 16-byte aligned before a call. Pushing an odd number of registers or missing alignment adjustment causes SSE instructions to fault.

Error 2: Wrong Argument Register

Passing a float in an integer register or vice versa causes silent data corruption. The ABI specifies separate register classes for integer and floating-point arguments.

Error 3: Missing Callee-Saved Save

If the callee modifies a callee-saved register without saving it, the caller's values are corrupted, causing subtle bugs that appear only after specific call chains.

Error 4: Incorrect Return Value Handling

Aggregate return values (structs larger than 16 bytes) are returned via a hidden pointer argument. Ignoring this convention causes stack corruption or incorrect results.

Error 5: Variadic Function Mismatch

Variadic functions (like printf) on x86-64 require AL to contain the number of vector register arguments. Failing to set AL correctly causes argument misalignment.

Practice Questions

Question 1

What is the difference between callee-saved and caller-saved registers?

Question 2

How does the System V AMD64 ABI pass the first six integer arguments?

Question 3

Why must the stack be 16-byte aligned before a call on x86-64?

Challenge

Implement a complete ABI code generator for a simple language with functions, struct parameters, and variadic arguments. Support both System V AMD64 and ARM AAPCS. Generate prologue, argument marshalling, call sequences, and epilogue.

FAQ

graph LR
    A[Function Signature] --> B[Argument Classification]
    B --> C[Register Assignment]
    B --> D[Stack Assignment]
    C --> E[Prologue Generation]
    D --> E
    E --> F[Call Sequence]
    F --> G[Epilogue Generation]
    style E fill:#2196F3,color:#fff,stroke-width:2px

Summary

Calling conventions define the binary protocol for function calls, specifying argument passing, register preservation, and stack management. The System V AMD64 and ARM AAPCS conventions use register-first strategies for performance, with clear distinctions between callee-saved and caller-saved registers. Correct ABI implementation is essential for compiler correctness and cross-language interoperability.

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.