System Calls — From User Space to Kernel — Complete Guide to OS Interfaces
In this tutorial, you'll learn about System Calls. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.
System calls are the controlled interface through which user-space programs request services from the operating system kernel — the only legal way to access hardware, create processes, or manage memory.
What You'll Learn & Why It Matters
In this tutorial, you'll learn how system calls work: the user-space to kernel-space transition, how the CPU switches privilege levels, how the syscall table dispatches requests, how strace captures every syscall your program makes, and how to write a custom syscall. You'll also learn the difference between Linux's syscall instruction and Windows' sysenter.
Real-world use: Every program you run makes hundreds of syscalls per second. When you type in a terminal, read() is called. When you save a file, write() and fsync() are called. When you allocate memory, brk() or mmap() is called. Durga Antivirus Pro uses ptrace() and fanotify() to intercept file operations for real-time scanning.
graph TD
subgraph "User Space (Ring 3)"
APP[Application
C / Python]
LIBC[libc / glibc]
end
subgraph "Transition"
SYSCALL[syscall instruction]
SAVE[Save registers]
SWITCH[Switch to kernel stack]
PUSH[Push syscall number + args]
end
subgraph "Kernel Space (Ring 0)"
TABLE[Syscall dispatch table]
HANDLER[Syscall handler function]
RESULT[Return result + errno]
end
APP -->|open, read, write| LIBC
LIBC -->|syscall| SYSCALL
SYSCALL --> SAVE --> SWITCH --> PUSH
PUSH --> TABLE
TABLE --> HANDLER
HANDLER --> RESULT
RESULT -->|Return value| APP
How a System Call Works
When a user-space program calls read(), the following happens:
- libc wrapper:
read()in libc moves arguments to registers - Syscall instruction:
syscall(x86-64) triggers a trap to kernel mode - Kernel entry:
entry_SYSCALL_64saves registers, switches stack - Dispatch: Kernel looks up the syscall number in
sys_call_table - Execute: The handler function (e.g.,
ksys_read) performs the operation - Return: Result placed in
rax, sysretq returns to user space
import ctypes
import struct
class SyscallTracer:
"""Simulate system call tracing (like strace)"""
syscall_names = {
0: 'read', 1: 'write', 2: 'open', 3: 'close',
9: 'mmap', 10: 'mprotect', 12: 'brk',
59: 'execve', 60: 'exit', 62: 'kill',
137: 'statfs', 257: 'openat',
}
def __init__(self):
self.calls = []
self.start_time = None
def syscall_enter(self, number, args):
name = self.syscall_names.get(number, f'syscall_{number}')
entry = {
'number': number,
'name': name,
'args': args,
'timestamp': 0,
}
self.calls.append(entry)
return entry
def syscall_exit(self, entry, result):
entry['result'] = result
return entry
def trace(self, func, *args):
"""Simulate tracing a function's syscalls"""
import time
self.start_time = time.time()
# Simulated syscalls for a file read operation
syscalls = [
(257, ('/etc/passwd', 0, 0)), "# openat
(0", (3, '0x7ffd...', 1024)), "# read
(1", (1, '0x7ffd...', 512)), "# write (echo to terminal)
(3", (3,)), # close
]
for num, args_list in syscalls:
entry = self.syscall_enter(num, args_list)
# Simulated return values
results = {257: 3, 0: 512, 1: 512, 3: 0}
result = results.get(num, 0)
self.syscall_exit(entry, result)
return self.report()
def report(self):
print(f'{"Syscall":<12} {"Args":<30} {"Result":<8}')
print('-' * 50)
for call in self.calls:
args_str = ', '.join(str(a) for a in call['args'][:2])
print(f'{call["name"]:<12} {args_str:<30} {call["result"]:<8}')
print(f'\nTotal syscalls: {len(self.calls)}')
tracer = SyscallTracer()
tracer.trace('cat', '/etc/passwd')
Expected output:
Syscall Args Result
--------------------------------------------------
openat /etc/passwd, 0, 0 3
read 3, 0x7ffd..., 1024 512
write 1, 0x7ffd..., 512 512
close 3 0
Total syscalls: 4
Tracing Syscalls with strace
strace is the most important tool for understanding what syscalls a program makes.
# Trace file operations for 'ls'
strace -e trace=file ls /tmp 2>&1 | head -15
# Trace network syscalls
strace -e trace=network curl https://example.com 2>&1 | head -10
# Count all syscalls by type
strace -c ls ~ 2>&1 | tail -20
# Trace a specific PID
strace -p 1234 -e trace=read,write
Expected output (file syscalls for ls):
execve("/usr/bin/ls", ["ls", "/tmp"], 0x7fff...) = 0
brk(NULL) = 0x555...
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\0\0\0\0\0\0"..., 832) = 832
close(3) = 0
newfstatat(AT_FDCWD, "/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}, 0) = 0
openat(AT_FDCWD, "/tmp", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
getdents64(3, 0x7ffd..., 32768) = 288
write(1, "file1.txt file2.txt ...\n", 24) = 24
close(3) = 0
Writing a Custom Syscall in a Kernel Module
Linux supports adding custom syscalls (for development kernels with CONFIG_DEBUG_KERNEL).
#include <linux/kernel.h>
#include <linux/syscalls.h>
#include <linux/uaccess.h>
#include <linux/slab.h>
/* Custom syscall: dodatech_hello
* Takes a user-space buffer and fills it with a greeting.
* Returns the length of the greeting string. */
SYSCALL_DEFINE2(dodatech_hello, char __user *, buf, size_t, len)
{
char *greeting = "Hello from DodaTech syscall!";
size_t msg_len = strlen(greeting) + 1; /* include null terminator */
int ret;
if (!buf || !len)
return -EINVAL;
/* Copy greeting to user space */
if (msg_len > len)
msg_len = len;
ret = copy_to_user(buf, greeting, msg_len);
if (ret)
return -EFAULT;
printk(KERN_INFO "dodatech_hello: sent '%s' to pid %d\n",
greeting, current->pid);
return msg_len;
}
Test it from user space:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/syscall.h>
/* Custom syscall number (dynamically allocated in this example) */
#define SYS_DODATECH_HELLO 450
int main() {
char buffer[128] = {0};
int ret;
/* Call the custom syscall */
ret = syscall(SYS_DODATECH_HELLO, buffer, sizeof(buffer));
if (ret < 0) {
perror("dodatech_hello");
return 1;
}
printf("Syscall returned %d bytes: %s\n", ret, buffer);
return 0;
}
Expected output:
Syscall returned 28 bytes: Hello from DodaTech syscall!
vDSO and vsyscall — Faster Syscalls
Some syscalls (like gettimeofday and clock_gettime) can be handled entirely in user space without a kernel transition, using the vDSO (virtual dynamic shared object).
#include <stdio.h>
#include <time.h>
#include <sys/time.h>
/* vDSO-accelerated syscalls: gettimeofday runs in user space */
int main() {
struct timespec ts1, ts2;
struct timeval tv;
/* These calls may be handled entirely in user space via vDSO */
clock_gettime(CLOCK_MONOTONIC, &ts1);
gettimeofday(&tv, NULL);
clock_gettime(CLOCK_MONOTONIC, &ts2);
long vdso_ns = (ts2.tv_sec - ts1.tv_sec) * 1000000000L +
(ts2.tv_nsec - ts1.tv_nsec);
printf("Realtime: %ld.%06ld\n", tv.tv_sec, tv.tv_usec);
printf("vDSO overhead (2 calls): %ld ns\n", vdso_ns);
return 0;
}
Expected output:
Realtime: 1719123456.789012
vDSO overhead (2 calls): 48 ns
# Check which syscalls are in vDSO
cat /proc/self/maps | grep vdso
# Dump the vDSO binary
dd if=/proc/self/mem of=/tmp/vdso.so bs=4096 skip=$((0x7fff...)) count=1 2>/dev/null || true
# Better: find vDSO symbol addresses
readelf -s /usr/lib/x86_64-linux-gnu/libvdso.so 2>/dev/null | head -10
# Show what strace can't trace (vDSO syscalls don't appear)
strace -c date 2>&1 | head -15
Expected output:
7fff...1000-7fff...2000 r-xp 00000000 00:00 0 [vdso]
System Call Overhead Benchmark
The cost of a system call (context switch to kernel mode and back) is significant compared to a regular function call.
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <time.h>
#include <sys/time.h>
#define NUM_ITERATIONS 1000000
static inline long nanoseconds(struct timespec *start, struct timespec *end) {
return (end->tv_sec - start->tv_sec) * 1000000000L +
(end->tv_nsec - start->tv_nsec);
}
int main() {
struct timespec start, end;
volatile int dummy;
long elapsed_ns;
/* Benchmark 1: No-op system call (getpid) */
clock_gettime(CLOCK_MONOTONIC, &start);
for (int i = 0; i < NUM_ITERATIONS; i++) {
dummy = getpid();
}
clock_gettime(CLOCK_MONOTONIC, &end);
elapsed_ns = nanoseconds(&start, &end);
printf("getpid syscall: %ld ns average (%d iterations)\n",
elapsed_ns / NUM_ITERATIONS, NUM_ITERATIONS);
/* Benchmark 2: No-op function call */
clock_gettime(CLOCK_MONOTONIC, &start);
for (int i = 0; i < NUM_ITERATIONS; i++) {
dummy = i;
}
clock_gettime(CLOCK_MONOTONIC, &end);
elapsed_ns = nanoseconds(&start, &end);
printf("function call: %ld ns average (%d iterations)\n",
elapsed_ns / NUM_ITERATIONS, NUM_ITERATIONS);
/* Benchmark 3: vDSO-accelerated clock_gettime */
clock_gettime(CLOCK_MONOTONIC, &start);
for (int i = 0; i < NUM_ITERATIONS; i++) {
clock_gettime(CLOCK_MONOTONIC, &end);
}
clock_gettime(CLOCK_MONOTONIC, &end);
elapsed_ns = nanoseconds(&start, &end);
printf("vDSO call: %ld ns average (%d iterations)\n",
elapsed_ns / NUM_ITERATIONS, NUM_ITERATIONS);
return 0;
}
Expected output:
getpid syscall: 127 ns average (1000000 iterations)
function call: 3 ns average (1000000 iterations)
vDSO call: 22 ns average (1000000 iterations)
Common Mistakes
1. Confusing Libc Functions with Syscalls
printf is not a system call — it's a libc wrapper that calls write() internally. Most standard library functions eventually make syscalls, but many (like strlen, atoi) never enter the kernel.
2. Forgetting to Check Syscall Return Values
Syscalls return -1 on error and set errno. Programs that ignore the return value silently continue after errors. Always check: if (ret == -1) { perror("open"); }.
3. Assuming Syscalls Are Expensive in All Cases
vDSO-accelerated syscalls (gettimeofday, clock_gettime) cost ~20ns — comparable to a function call. Only "real" syscalls (file I/O, networking, Process creation) have the full 100-200ns overhead.
4. Not Using Batched Syscalls
Some syscalls support batching. readv/writev perform scatter/gather I/O with one syscall instead of multiple read/write calls. sendmmsg sends multiple messages in one syscall.
5. Confusing Syscall Numbers Across Architectures
Syscall numbers differ between x86-64 and ARM64. syscall __NR_read works on all architectures; hardcoded numbers break on different CPUs. Always use the <sys/syscall.h> macros.
Practice Questions
1. What happens when a program calls a system call?
The program calls a libc wrapper, which executes the syscall instruction. The CPU switches from ring 3 to ring 0, the kernel saves registers, dispatches the syscall by number, executes the handler, places the result in rax, and returns to user space via sysretq.
2. Why does strace not show vDSO syscalls?
vDSO handlers execute entirely in user space using kernel-mapped memory pages. They never trigger the syscall instruction, so strace (which intercepts the syscall entry point) never sees them. They are invisible to strace.
3. What is the difference between syscall and int 0x80 on x86?
int 0x80 is the legacy 32-bit syscall mechanism (slow, uses interrupt gate). syscall is the 64-bit fast syscall entry (uses MSRs, no interrupt table lookup). syscall is ~3x faster. On modern x86-64, all syscalls use syscall/sysretq.
4. Challenge: Write a Python script using ctypes that calls the getpid() syscall directly (bypassing libc) using Python's ctypes.CDLL with syscall(39) on x86-64. Compare its speed to os.getpid().
5. Real-World Task: Run strace -c ls /tmp and identify which syscall is called most frequently. Then run the same with strace -e read,write,openat,close to see just the file I/O syscalls. Explain why brk and mmap appear.
Mini Project: Syscall Latency Explorer
import ctypes
import os
import time
import statistics
class SyscallLatencyExplorer:
def __init__(self):
self.libc = ctypes.CDLL('libc.so.6')
self.results = {}
def measure_syscall(self, name, syscall_fn, iterations=100000):
times = []
for _ in range(iterations):
start = time.perf_counter_ns()
syscall_fn()
end = time.perf_counter_ns()
times.append(end - start)
avg = statistics.mean(times) / 1000 # microseconds
p99 = sorted(times)[int(len(times) * 0.99)] / 1000
min_val = min(times) / 1000
max_val = max(times) / 1000
self.results[name] = {
'avg_us': avg,
'p99_us': p99,
'min_us': min_val,
'max_us': max_val,
}
return self.results[name]
def benchmark(self):
print('Syscall Latency Benchmark (microseconds)\n')
# getpid — classic syscall
self.measure_syscall('getpid', os.getpid)
# clock_gettime — vDSO accelerated
t = time.clock_gettime_ns # noqa
def clock_gettime_call():
time.clock_gettime_ns(time.CLOCK_MONOTONIC)
self.measure_syscall('clock_gettime (vDSO)', clock_gettime_call)
# Open and close a file
fd = os.open('/dev/null', os.O_RDONLY)
def read_call():
os.read(fd, 1)
self.measure_syscall('read', read_call, iterations=50000)
os.close(fd)
# Write to stdout
def write_call():
os.write(1, b'x')
self.measure_syscall('write', write_call, iterations=50000)
print(f'{"Syscall":<25} {"Avg (us)":<10} {"P99 (us)":<10} '
f'{"Min (us)":<10} {"Max (us)":<10}')
print('-' * 65)
for name, stats in sorted(self.results.items(),
key=lambda x: x[1]['avg_us']):
print(f'{name:<25} {stats["avg_us"]:<10.2f} '
f'{stats["p99_us"]:<10.2f} '
f'{stats["min_us"]:<10.2f} '
f'{stats["max_us"]:<10.2f}')
explorer = SyscallLatencyExplorer()
explorer.benchmark()
Expected output:
Syscall Latency Benchmark (microseconds)
Syscall Avg (us) P99 (us) Min (us) Max (us)
-----------------------------------------------------------------
clock_gettime (vDSO) 0.02 0.05 0.01 1.20
getpid 0.13 0.18 0.11 2.50
read 0.15 0.22 0.12 3.10
write 0.18 0.30 0.13 4.50
FAQ
Related Concepts
What's Next
You now understand system calls from user space to kernel. Next, learn about the Linux boot process to see how the kernel loads and initializes the system, or explore kernel modules for adding code to the running kernel.
- Practice daily — Run
strace -c lsand identify the most frequently called syscall inls. - Build a project — Create a mini strace clone in Python using
ptraceto intercept syscalls. - Explore related topics — Study seccomp profiles for hardening containerized applications.
Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro