C I/O Multiplexing — select, poll, and epoll for Concurrent I/O
In this tutorial, you will learn about C I/O Multiplexing. We cover key concepts, practical examples, and best practices to help you master this topic.
C I/O multiplexing enables a single thread to monitor multiple file descriptors (sockets, pipes, files) for readiness to read, write, or error events using system calls like select, poll, and epoll, forming the foundation of event-driven network servers.
What You Will Learn
- Monitoring multiple file descriptors with select()
- Scalable I/O with poll()
- High-performance I/O with epoll (Linux)
- Edge-triggered vs level-triggered modes
- Building an event loop
- Handling read, write, and error events
- Performance characteristics of each API
Why It Matters
Thread-per-client or Process-per-client approaches do not scale beyond a few hundred connections. Each thread consumes stack memory (typically 8 MB) and context switching overhead becomes significant. I/O multiplexing lets one thread handle thousands of connections efficiently. High-performance servers like Nginx, Redis, and Node.js use event-driven architectures built on epoll/kqueue. Durga Antivirus Pro's real-time file scanner uses inotify (similar multiplexing) to monitor all file system events without polling.
Real-World Use
A Redis server handles 100,000+ client connections on a single thread. It does not use threads per client. Instead, it uses an event loop with epoll that monitors all client sockets. When a client sends a command, epoll wakes up, Redis reads the command, processes it, sends the response, and returns to epoll_wait.
Learning Path
flowchart LR A[Network Sockets] --> B[I/O Multiplexing\nYou are here] B --> C[Project Calculator] style B fill:#f90,color:#fff
select() — Monitor Multiple Descriptors
#include <stdio.h>
#include <stdlib.h>
#include <sys/select.h>
#include <sys/time.h>
#include <unistd.h>
#include <fcntl.h>
int main() {
int fds[3];
char buffer[1024];
// Open /dev/zero which always yields bytes
fds[0] = open("/dev/zero", O_RDONLY);
// Open stdin for reading
fds[1] = STDIN_FILENO;
while (1) {
fd_set read_fds;
FD_ZERO(&read_fds);
FD_SET(fds[0], &read_fds);
FD_SET(fds[1], &read_fds);
int maxfd = (fds[0] > fds[1]) ? fds[0] : fds[1];
struct timeval timeout = {5, 0}; // 5 seconds
printf("Waiting for data...\n");
int ret = select(maxfd + 1, &read_fds, NULL, NULL, &timeout);
if (ret == -1) {
perror("select");
break;
} else if (ret == 0) {
printf("Timeout! No data in 5 seconds.\n");
break;
}
if (FD_ISSET(fds[0], &read_fds)) {
int n = read(fds[0], buffer, sizeof(buffer));
printf("Read %d bytes from /dev/zero\n", n);
}
if (FD_ISSET(fds[1], &read_fds)) {
int n = read(fds[1], buffer, sizeof(buffer) - 1);
if (n > 0) {
buffer[n] = '\0';
printf("stdin: %s", buffer);
}
}
}
close(fds[0]);
return 0;
}
select()-Based TCP Server
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/select.h>
#include <sys/socket.h>
#include <netinet/in.h>
#define PORT 8080
#define MAX_CLIENTS 10
#define BUFFER_SIZE 1024
int main() {
int server_fd, client_fds[MAX_CLIENTS];
struct sockaddr_in address;
int opt = 1;
socklen_t addrlen = sizeof(address);
// Create server socket
server_fd = socket(AF_INET, SOCK_STREAM, 0);
setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
address.sin_family = AF_INET;
address.sin_addr.s_addr = INADDR_ANY;
address.sin_port = htons(PORT);
bind(server_fd, (struct sockaddr*)&address, sizeof(address));
listen(server_fd, 3);
for (int i = 0; i < MAX_CLIENTS; i++) {
client_fds[i] = -1;
}
printf("select() server on port %d\n", PORT);
while (1) {
fd_set read_fds;
FD_ZERO(&read_fds);
FD_SET(server_fd, &read_fds);
int maxfd = server_fd;
for (int i = 0; i < MAX_CLIENTS; i++) {
if (client_fds[i] > 0) {
FD_SET(client_fds[i], &read_fds);
if (client_fds[i] > maxfd) {
maxfd = client_fds[i];
}
}
}
int activity = select(maxfd + 1, &read_fds, NULL, NULL, NULL);
if (activity < 0) {
perror("select");
break;
}
// New connection
if (FD_ISSET(server_fd, &read_fds)) {
int new_socket = accept(server_fd, NULL, NULL);
printf("New client connected\n");
for (int i = 0; i < MAX_CLIENTS; i++) {
if (client_fds[i] == -1) {
client_fds[i] = new_socket;
break;
}
}
}
// Client activity
for (int i = 0; i < MAX_CLIENTS; i++) {
int sd = client_fds[i];
if (FD_ISSET(sd, &read_fds)) {
char buffer[BUFFER_SIZE] = {0};
int valread = read(sd, buffer, BUFFER_SIZE - 1);
if (valread == 0) {
// Client disconnected
printf("Client disconnected\n");
close(sd);
client_fds[i] = -1;
} else {
buffer[valread] = '\0';
printf("Received: %s", buffer);
send(sd, buffer, valread, 0);
}
}
}
}
close(server_fd);
return 0;
}
poll() — More Scalable Than select
poll() uses an array of struct pollfd instead of bit masks, removing the FD_SETSIZE limit:
#include <stdio.h>
#include <stdlib.h>
#include <poll.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#define PORT 8081
#define MAX_CLIENTS 100
#define BUFFER_SIZE 1024
int main() {
int server_fd;
struct sockaddr_in address;
int opt = 1;
server_fd = socket(AF_INET, SOCK_STREAM, 0);
setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
address.sin_family = AF_INET;
address.sin_addr.s_addr = INADDR_ANY;
address.sin_port = htons(PORT);
bind(server_fd, (struct sockaddr*)&address, sizeof(address));
listen(server_fd, 3);
struct pollfd fds[MAX_CLIENTS];
fds[0].fd = server_fd;
fds[0].events = POLLIN;
int nfds = 1;
for (int i = 1; i < MAX_CLIENTS; i++) {
fds[i].fd = -1;
}
printf("poll() server on port %d\n", PORT);
while (1) {
int ret = poll(fds, nfds, -1);
if (ret < 0) {
perror("poll");
break;
}
// New connection
if (fds[0].revents & POLLIN) {
int new_socket = accept(server_fd, NULL, NULL);
printf("New client connected\n");
for (int i = 1; i < MAX_CLIENTS; i++) {
if (fds[i].fd == -1) {
fds[i].fd = new_socket;
fds[i].events = POLLIN;
nfds = (i + 1 > nfds) ? i + 1 : nfds;
break;
}
}
}
// Client data
for (int i = 1; i < nfds; i++) {
if (fds[i].fd == -1) continue;
if (fds[i].revents & POLLIN) {
char buffer[BUFFER_SIZE] = {0};
int valread = read(fds[i].fd, buffer, BUFFER_SIZE - 1);
if (valread == 0) {
printf("Client disconnected\n");
close(fds[i].fd);
fds[i].fd = -1;
} else {
buffer[valread] = '\0';
printf("Received: %s", buffer);
send(fds[i].fd, buffer, valread, 0);
}
}
}
}
close(server_fd);
return 0;
}
epoll — Linux High-Performance I/O (Level-Triggered)
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/epoll.h>
#include <sys/socket.h>
#include <netinet/in.h>
#define PORT 8082
#define MAX_EVENTS 100
#define BUFFER_SIZE 1024
int main() {
int server_fd;
struct sockaddr_in address;
int opt = 1;
server_fd = socket(AF_INET, SOCK_STREAM, 0);
setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
address.sin_family = AF_INET;
address.sin_addr.s_addr = INADDR_ANY;
address.sin_port = htons(PORT);
bind(server_fd, (struct sockaddr*)&address, sizeof(address));
listen(server_fd, 3);
// Create epoll instance
int epoll_fd = epoll_create1(0);
if (epoll_fd < 0) {
perror("epoll_create1");
exit(1);
}
// Add server socket to epoll
struct epoll_event ev;
ev.events = EPOLLIN;
ev.data.fd = server_fd;
epoll_ctl(epoll_fd, EPOLL_CTL_ADD, server_fd, &ev);
struct epoll_event events[MAX_EVENTS];
printf("epoll server on port %d\n", PORT);
while (1) {
int n = epoll_wait(epoll_fd, events, MAX_EVENTS, -1);
for (int i = 0; i < n; i++) {
if (events[i].data.fd == server_fd) {
// New connection
int client_fd = accept(server_fd, NULL, NULL);
printf("New client connected\n");
// Add client to epoll
ev.events = EPOLLIN;
ev.data.fd = client_fd;
epoll_ctl(epoll_fd, EPOLL_CTL_ADD, client_fd, &ev);
} else {
// Client data
char buffer[BUFFER_SIZE] = {0};
int valread = read(events[i].data.fd, buffer, BUFFER_SIZE - 1);
if (valread == 0) {
printf("Client disconnected\n");
close(events[i].data.fd);
epoll_ctl(epoll_fd, EPOLL_CTL_DEL, events[i].data.fd, NULL);
} else {
buffer[valread] = '\0';
printf("Received: %s", buffer);
send(events[i].data.fd, buffer, valread, 0);
}
}
}
}
close(epoll_fd);
close(server_fd);
return 0;
}
epoll Edge-Triggered Mode
Edge-triggered mode notifies only when the state changes (e.g., data arrives after the buffer was empty):
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <string.h>
#include <errno.h>
#include <sys/epoll.h>
void set_nonblocking(int fd) {
int flags = fcntl(fd, F_GETFL, 0);
fcntl(fd, F_SETFL, flags | O_NONBLOCK);
}
int main() {
int epoll_fd = epoll_create1(0);
struct epoll_event ev;
// Monitor stdin in edge-triggered mode
ev.events = EPOLLIN | EPOLLET; // Edge-triggered
ev.data.fd = STDIN_FILENO;
epoll_ctl(epoll_fd, EPOLL_CTL_ADD, STDIN_FILENO, &ev);
set_nonblocking(STDIN_FILENO);
struct epoll_event events[10];
char buffer[1024];
printf("Edge-triggered stdin monitor (type something):\n");
while (1) {
int n = epoll_wait(epoll_fd, events, 10, 5000);
if (n == 0) {
printf("Five-second timeout\n");
break;
}
for (int i = 0; i < n; i++) {
if (events[i].data.fd == STDIN_FILENO) {
// Must read until EAGAIN in edge-triggered mode
while (1) {
int ret = read(STDIN_FILENO, buffer, sizeof(buffer) - 1);
if (ret > 0) {
buffer[ret] = '\0';
printf("Read: %s", buffer);
} else if (ret == 0) {
printf("EOF\n");
return 0;
} else if (errno == EAGAIN) {
break; // No more data
} else {
perror("read");
return 1;
}
}
}
}
}
close(epoll_fd);
return 0;
}
Performance Comparison
| Method | File Descriptor Limit | Setup Overhead | Per-Event Overhead | Platform |
|---|---|---|---|---|
| select | FD_SETSIZE (1024) | Low | O(n) scan all FDs | All POSIX |
| poll | No hard limit (memory) | Low | O(n) scan all FDs | All POSIX |
| epoll | No hard limit | Medium (epoll_create) | O(1) active events | Linux |
| kqueue | No hard limit | Medium | O(1) active events | BSD/macOS |
Common Mistakes
Not resetting fd_set before each select() call: select() modifies the fd_set argument. You must reinitialize with FD_ZERO and FD_SET before every call.
Exceeding FD_SETSIZE with select(): select() cannot monitor more than FD_SETSIZE file descriptors (typically 1024). Use poll() or epoll() for large numbers of connections.
Not draining the socket in edge-triggered epoll: In EPOLLET mode, you must read until recv() returns EAGAIN. If you stop early, you miss data and never get notified again.
Forgetting the listen socket in epoll: The server's listen socket must be added to epoll too. New connections arrive as EPOLLIN events on the server socket.
Using EPOLLONESHOT incorrectly: After an EPOLLONESHOT event fires, the fd is removed from epoll. You must re-arm it with epoll_ctl. Use this for multi-threaded epoll to avoid thundering herd.
Assuming select/poll work with regular files: select() and poll() always report regular files as readable and writable. They are designed for sockets, pipes, and terminals.
Blocking in edge-triggered mode without non-blocking I/O: Edge-triggered epoll requires non-blocking sockets. If a socket blocks during read/write, your event loop hangs.
Practice Questions
- What is the difference between level-triggered and edge-triggered event notification?
- Why does select() need the
maxfd + 1parameter? - How does poll() solve select()'s FD_SETSIZE limitation?
- Why must you read until EAGAIN in edge-triggered epoll?
- Challenge: Implement a simple event-driven HTTP server using epoll that serves static files. Use non-blocking sockets and edge-triggered epoll. Handle partial writes (EPOLLOUT events). Test with 1000 concurrent connections using a tool like
wrkorab.
Mini Project
Build an event-driven chat server using epoll:
- Single-threaded, no mutexes needed
- Uses epoll edge-triggered mode
- Maintains a Linked List of connected clients
- Clients send JSON messages:
{"type": "message", "sender": "alice", "text": "hello"} - Broadcasts messages to all other connected clients
- Handles partial reads properly (buffers per connection)
- Handles partial writes (registers EPOLLOUT when send buffer is non-empty)
- Supports up to 10,000 concurrent connections
- Gracefully handles client disconnects (EPOLLRDHUP or recv=0)
FAQ
What is Next
Proceed to Project Calculator to build a command-line calculator that uses the skills you have learned. Then continue with Project File Splitter and other project lessons.