Assembly System Calls — Linux Kernel Interface

In this tutorial, you will learn about Assembly System Calls. We cover key concepts, practical examples, and best practices to help you master this topic.

Linux system calls in assembly use the syscall instruction with syscall numbers in RAX and arguments in RDI, RSI, RDX, R10, R8, R9 for kernel operations.

What You'll Learn

• The syscall instruction
• Syscall number table
• Passing arguments to the kernel
• Common syscalls: read, write, exit, mmap

Why It Matters

Syscalls are the only way to interact with the OS (files, network, processes). Durga Antivirus Pro uses syscalls directly in its performance-critical file scanning engine to bypass libc overhead.

Real-World Use

Operating system programming, Embedded Systems without libc, performance-critical I/O, and security tools that need direct kernel access.

flowchart LR
    A["User Program"] --> B["syscall"]
    B --> C["Kernel"]
    C --> D["Service Handler"]
    D --> E["Return"]
    F["RAX = number"] --> B
    G["RDI, RSI, RDX"] --> B
    style A fill:#2563eb,stroke:#2563eb,color:#fff
    style B fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style C fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style D fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style E fill:#dbeafe,stroke:#2563eb,color:#1e40af

The syscall Instruction

; syscall — enter kernel mode
; RAX = syscall number
; RDI = arg1, RSI = arg2, RDX = arg3
; R10 = arg4, R8 = arg5, R9 = arg6
; Return value in RAX

mov rax, 60     ; syscall number for exit
xor rdi, rdi    ; exit code 0
syscall         ; never returns

Common Syscall Numbers

Writing to stdout

section .data
    msg db "Hello via syscall!", 10
    len equ $ - msg

section .text
    global _start

_start:
    ; write(1, msg, len)
    mov rax, 1      ; syscall: write
    mov rdi, 1      ; fd: stdout
    mov rsi, msg    ; buffer
    mov rdx, len    ; length
    syscall

    ; exit(0)
    mov rax, 60
    xor rdi, rdi
    syscall

Reading from stdin

section .bss
    buf resb 64

section .text
    global _start

_start:
    ; read(0, buf, 64)
    mov rax, 0      ; syscall: read
    mov rdi, 0      ; fd: stdin
    mov rsi, buf
    mov rdx, 64
    syscall

    ; rax = number of bytes read
    ; write(1, buf, rax)
    mov rdx, rax
    mov rax, 1
    mov rdi, 1
    mov rsi, buf
    syscall

    ; exit(0)
    mov rax, 60
    xor rdi, rdi
    syscall

Opening and Reading a File

section .data
    filename db "test.txt", 0
    flags dq 0        ; O_RDONLY = 0

section .bss
    buf resb 256

section .text
    global _start

_start:
    ; open(filename, O_RDONLY)
    mov rax, 2
    mov rdi, filename
    xor rsi, rsi      ; flags = 0 (O_RDONLY)
    xor rdx, rdx      ; mode = 0
    syscall

    ; rax = file descriptor (or negative error)
    mov r8, rax       ; save fd

    ; read(fd, buf, 256)
    mov rax, 0
    mov rdi, r8
    mov rsi, buf
    mov rdx, 256
    syscall

    ; write(1, buf, rax)
    mov rdx, rax
    mov rax, 1
    mov rdi, 1
    mov rsi, buf
    syscall

    ; close(fd)
    mov rax, 3
    mov rdi, r8
    syscall

    ; exit(0)
    mov rax, 60
    xor rdi, rdi
    syscall

mmap for Memory Allocation

; mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
mov rax, 9          ; syscall: mmap
xor rdi, rdi        ; addr = NULL (kernel chooses)
mov rsi, 4096       ; length
mov rdx, 3          ; prot = PROT_READ | PROT_WRITE
mov r10, 0x22       ; flags = MAP_PRIVATE | MAP_ANONYMOUS
mov r8, -1          ; fd = -1 (none)
xor r9, r9          ; offset = 0
syscall

; rax = allocated memory address
mov rdi, rax        ; use the memory
mov byte [rdi], 'A'

Error Handling

_start:
    mov rax, 2      ; open
    mov rdi, filename
    xor rsi, rsi
    xor rdx, rdx
    syscall

    ; Negative return means error
    test rax, rax
    js error_handler

    ; Use fd in rax...

error_handler:
    ; rax contains -errno (negative)
    neg rax          ; positive errno value
    ; handle error...
    mov rax, 60
    mov rdi, 1
    syscall

Common Mistakes

1. Using R10 instead of RCX for arg4

The kernel uses R10 for the 4th argument because the syscall instruction clobbers RCX (saves RIP).

2. Not checking return values

Syscalls return -errno in RAX on failure (e.g., -2 for ENOENT). Always check with test rax, rax; js error.

3. Wrong syscall number

Syscall numbers differ between architectures (x86-64 vs ARM64 vs x86-32). Always verify for your target.

4. Missing buffer space

read/write with too-large count can overflow buffers. Always pass the actual buffer size.

5. Forgetting to restore RSP after mmap

If mmap replaces the stack, the old RSP becomes invalid. Only advanced users should map near the stack.

Practice Questions

1. What register holds the syscall number?

RAX.

2. How does the kernel return errors from syscalls?

Returns a negative errno value in RAX. -2 means ENOENT (file not found).

3. Which argument register does the 4th syscall argument use?

R10, not RCX. The syscall instruction clobbers RCX with the return RIP.

4. What is the syscall number for exit?

60 on x86-64 Linux.

Challenge: Write a program that copies a file using only syscalls (open, read, write, close).

section .data
    src_file db "source.txt", 0
    dst_file db "dest.txt", 0

section .bss
    buffer resb 4096

section .text
    global _start

_start:
    mov rax, 2
    mov rdi, src_file
    xor rsi, rsi
    syscall
    mov r8, rax

    mov rax, 2
    mov rdi, dst_file
    mov rsi, 0x41     ; O_WRONLY | O_CREAT
    mov rdx, 0x1A4    ; mode 644 octal
    syscall
    mov r9, rax

copy_loop:
    mov rax, 0
    mov rdi, r8
    mov rsi, buffer
    mov rdx, 4096
    syscall
    test rax, rax
    jle done

    mov rdx, rax
    mov rax, 1
    mov rdi, r9
    mov rsi, buffer
    syscall
    jmp copy_loop

done:
    mov rax, 3
    mov rdi, r8
    syscall
    mov rax, 3
    mov rdi, r9
    syscall
    mov rax, 60
    xor rdi, rdi
    syscall

FAQ

{{< faq question="What is the difference between syscall and int 0x80?" >}} syscall is the modern 64-bit instruction (faster). int 0x80 is the legacy 32-bit mechanism (slower, uses different syscall numbers). {{< /faq >}}

{{< faq question="Can I use C library functions instead of syscalls?" >}} Yes. Linking with libc lets you call printf, fopen, etc. Direct syscalls are for when you need minimal dependencies. {{< /faq >}}

{{< faq question="Do syscalls preserve all registers?" >}} Most registers are preserved except RAX (return value), RCX (saved RIP), and R11 (saved RFLAGS). {{< /faq >}}

{{< faq question="What happens if a syscall is interrupted?" >> The syscall returns -EINTR. You should retry interrupted syscalls in a loop. {{< /faq >}}

{{< faq question="How do I find syscall numbers for my architecture?" >}} Check /usr/include/x86_64-linux-gnu/asm/unistd_64.h for x86-64 or asm/unistd.h for 32-bit. {{< /faq >}}

Mini Project

Write a program that counts the number of bytes in a file using only syscalls.

section .data
    filename db "data.txt", 0

section .text
    global _start

_start:
    mov rax, 2
    mov rdi, filename
    xor rsi, rsi
    syscall
    mov r12, rax

    xor r13, r13

count_loop:
    sub rsp, 1024
    mov rax, 0
    mov rdi, r12
    mov rsi, rsp
    mov rdx, 1024
    syscall
    test rax, rax
    jle done
    add r13, rax
    add rsp, 1024
    jmp count_loop

done:
    add rsp, 1024
    mov rax, 3
    mov rdi, r12
    syscall

    mov rdi, r13
    mov rax, 60
    syscall

What's Next

Now that you understand system calls, proceed to string operations.