Assembly Guide — Stack Operations and Function Calls

In this tutorial, you will learn about Assembly Guide. We cover key concepts, practical examples, and best practices to help you master this topic.

The x86-64 stack is a last-in-first-out (LIFO) data structure pointed to by RSP, used for function call management, local variable storage, and temporary data holds.

What You'll Learn

• Stack growth and alignment
• Push and pop operations
• Function prologue and epilogue
• Calling conventions and parameter passing
• Stack frames and local variables

Why It Matters

Stack understanding is fundamental to function calls, Recursion, and debugging. Durga Antivirus Pro analyzes stack frames during malware reverse engineering.

Real-World Use

Function call management, local variable allocation, Exception Handling, and context switching.

flowchart LR
    A["Stack"] --> B["Push/Pop"]
    B --> C["Frames"]
    C --> D["Call/Ret"]
    D --> E["Parameters"]
    A:::current --> B
    style A fill:#2563eb,stroke:#2563eb,color:#fff
    style B fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style C fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style D fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style E fill:#f1f5f9,stroke:#94a3b8,color:#64748b

Push and Pop

; Push: decrement RSP, store value
push rax    ; RSP -= 8, [RSP] = RAX
push 100    ; RSP -= 8, [RSP] = 100

; Pop: load value, increment RSP
pop rax     ; RAX = [RSP], RSP += 8

; Push/pop preserves values across calls
push rbx    ; Save RBX
push rcx    ; Save RCX
; ... use RBX and RCX ...
pop rcx     ; Restore RCX
pop rbx     ; Restore RBX

Function Call

; The call instruction:
; 1. Pushes return address (RIP after call) onto stack
; 2. Jumps to function address

call myfunc   ; Push return address, jump to myfunc

; myfunc executes...
; At the end:
ret           ; Pop return address, jump back

; The ret instruction pops the return address into RIP

Function Prologue

; Standard function prologue
myfunc:
    push rbp           ; Save caller's RBP
    mov rbp, rsp       ; Set our frame pointer
    sub rsp, 32        ; Allocate local variables (32 bytes)

    ; Now:
    ; RBP points to saved RBP
    ; Parameters: [RBP + 16], [RBP + 24], etc.
    ; Return address: [RBP + 8]
    ; Saved RBP: [RBP]
    ; Local vars: [RBP - 8], [RBP - 16], etc.

Function Epilogue

; Standard function epilogue
myfunc:
    ; ... function body ...
    mov rsp, rbp       ; Restore stack pointer
    pop rbp            ; Restore caller's RBP
    ret                ; Return to caller

; Or simply (if no local stack allocation):
    pop rbp
    ret

Parameter Passing

; System V AMD64 ABI (Linux):
; First 6 params: RDI, RSI, RDX, RCX, R8, R9
; Additional params on stack (right to left)

; Example: func(a, b, c, d, e, f, g)
; a -> RDI, b -> RSI, c -> RDX
; d -> RCX, e -> R8, f -> R9
; g -> [RSP + 8] (after call, before prologue)

; After prologue, stack params at:
; [RBP + 16] - first stack param (g)
; [RBP + 24] - second stack param

Local Variables

myfunc:
    push rbp
    mov rbp, rsp
    sub rsp, 32       ; Allocate 32 bytes for locals

    ; Local variables:
    ; [RBP - 8]  - local1 (8 bytes)
    ; [RBP - 16] - local2 (8 bytes)
    ; [RBP - 24] - local3 (8 bytes)

    mov rax, 42
    mov [rbp - 8], rax  ; local1 = 42

    ; Use local
    mov rax, [rbp - 8]
    add rax, 10

    mov rsp, rbp
    pop rbp
    ret

Stack Alignment

; ABI requires 16-byte stack alignment at call
; RSP % 16 == 0 before call instruction

; Example of ensuring alignment:
myfunc:
    push rbp           ; RSP -= 8 (now misaligned)
    mov rbp, rsp
    sub rsp, 8         ; RSP -= 8 (now 16-byte aligned)
    ; or
    and spl, -16       ; Force 16-byte alignment

Common Mistakes

1. Stack imbalance

Every push must have a matching pop. Mismatch causes corrupted return addresses.

2. Forgetting alignment

Calling functions with misaligned stack crashes in SSE/AVX code (movaps requires alignment).

3. Return address corruption

Writing beyond local variable space overwrites the saved return address.

4. Wrong parameter location

Stack parameters start at [RBP+16], not [RBP+8] (which is the return address).

5. Not preserving red zone

The 128 bytes below RSP (red zone) may be overwritten by signal handlers. Don't use it for persistent data.

Practice Questions

1. What happens during a call instruction? The return address (current RIP + length of call) is pushed onto the stack, then execution jumps to the target.

2. Why save RBP in the prologue? To restore the caller's frame pointer on return. RBP allows accessing parameters and locals through a fixed base.

3. How is the stack aligned? The ABI requires 16-byte alignment before call. After push RBP, RSP is 8-byte aligned. Functions typically subtract an odd multiple of 8.

Challenge: Write an assembly function that takes three arguments and returns their sum, following the calling convention.

FAQ

{{< faq question="What direction does the stack grow?" >} The stack grows downward (toward lower addresses). RSP decreases on push. {{< /faq >}}

{{< faq question="What is the red zone?" >} The 128 bytes below RSP that can be used by leaf functions (functions that don't call others) without adjusting RSP. {{< /faq >}}

{{< faq question="Can I use RBP as a general register?" >} Yes, with optimization flags. GCC uses -fomit-frame-pointer to use RBP as a GPR. {{< /faq >}}

{{< faq question="What is a stack frame?" >} The region on the stack belonging to a function, bounded by RBP (or RSP at entry) and the current RSP. {{< /faq >}}

{{< faq question="How large can the stack be?" >} Stack size is limited (typically 8MB on Linux, 1MB on Windows). Use heap for large allocations. {{< /faq >}}

Mini Project

Write a function with proper prologue and epilogue:

; long max(long a, long b, long c)
; Args: RDI=a, RSI=b, RDX=c
; Return: RAX = max(a, b, c)

max:
    push rbp
    mov rbp, rsp

    mov rax, rdi       ; rax = a
    cmp rax, rsi       ; compare a, b
    cmovl rax, rsi     ; if a < b, rax = b
    cmp rax, rdx       ; compare current max, c
    cmovl rax, rdx     ; if max < c, rax = c

    pop rbp
    ret

What's Next

Now that you understand the stack, explore the x86-64 instruction set in detail.