VPN Setup & Configuration -- WireGuard, OpenVPN & Protocols

In this tutorial, you'll learn about VPN Setup & Configuration. We cover key concepts, practical examples, and best practices.

A Virtual Private Network (VPN) encrypts your internet traffic and routes it through a remote server, hiding your IP and protecting data from eavesdroppers on untrusted networks like public Wi-Fi.

What You'll Learn

You will learn how to set up WireGuard and OpenVPN, understand VPN protocols, configure server and client endpoints, and apply security best practices for production deployments.

Why It Matters

VPN usage has grown 27% year over year as remote work and public cloud access become the norm. Every organization needs secure encrypted tunnels for employees, partners, and automated systems.

Real-World Use

A DevOps team deploys a WireGuard mesh across three cloud regions so database replicas communicate without exposing traffic to the public internet, reducing the attack surface by 90%.

VPN Protocols Compared

WireGuard Setup

WireGuard is a modern VPN protocol with a lean 4,000-line codebase, in-kernel performance, and built-in roaming. It uses ChaCha20-Poly1305 for encryption and Curve25519 for key exchange -- both resistant to quantum computing advances compared to older algorithms.

Server Configuration

The server acts as the hub that routes traffic between connected peers and the internet.

[Interface]
Address = 10.0.0.1/24
PrivateKey = <server-private-key>
ListenPort = 51820

[Peer]
PublicKey = <client-public-key>
AllowedIPs = 10.0.0.2/32

Expected behavior: WireGuard accepts encrypted packets from the peer with the matching public key and assigns the IP 10.0.0.2.

How WireGuard Handles Encryption Differently

Unlike OpenVPN which relies on OpenSSL for cryptographic operations, WireGuard implements its own crypto layer using only five algorithms: Curve25519, ChaCha20, Poly1305, BLAKE2s, and HKDF. This minimal dependency surface reduces the attack surface significantly. Each packet is encrypted with an ephemeral session key derived from the long-term static keys, providing perfect forward secrecy -- if a static key is compromised, past sessions remain secure.

Generate Keys

wg genkey | tee server-private.key | wg pubkey > server-public.key
wg genkey | tee client-private.key | wg pubkey > client-public.key

Expected output: Four files containing the respective base64-encoded keys.

Client Configuration

[Interface]
Address = 10.0.0.2/24
PrivateKey = <client-private-key>
DNS = 1.1.1.1

[Peer]
PublicKey = <server-public-key>
Endpoint = your-server.com:51820
AllowedIPs = 0.0.0.0/0

Expected behavior: All traffic routes through the VPN tunnel. DNS queries resolve through Cloudflare.

OpenVPN Setup

OpenVPN offers enterprise-grade features including certificate-based auth, TCP/UDP fallback, and extensive auditing.

Server Configuration

port 1194
proto udp
dev tun
ca ca.crt
cert server.crt
key server.key
dh dh.pem
server 10.8.0.0 255.255.255.0
push "redirect-gateway def1"
push "dhcp-option DNS 1.1.1.1"
keepalive 10 120
cipher AES-256-GCM
auth SHA256
user nobody
group nogroup
status openvpn-status.log
verb 3

Expected behavior: OpenVPN listens on UDP 1194, assigns clients IPs from 10.8.0.0/24, and redirects all traffic through the tunnel.

Generate Certificates with EasyRSA

./easyrsa init-pki
./easyrsa build-ca nopass
./easyrsa gen-req server nopass
./easyrsa sign-req server server
./easyrsa gen-dh
openvpn --genkey secret ta.key

Expected output: A complete PKI hierarchy under pki/ with CA, server cert, and DH params.

Kill Switch Configuration

A kill switch blocks all internet traffic if the VPN connection drops, preventing data leaks.

#!/bin/bash
# WireGuard kill switch using nftables
nft add table inet killswitch
nft add chain inet killswitch prerouting '{ type filter hook prerouting priority 0; policy accept; }'
nft add rule inet killswitch prerouting iif != wg0 meta l4proto { tcp, udp } drop

Expected behavior: If the WireGuard interface goes down, all non-VPN network traffic is immediately dropped. Only traffic through wg0 is allowed.

Performance Tuning

WireGuard performs close to native line speed on modern hardware. Tune these parameters for optimal throughput.

# Increase UDP receive buffer for high-bandwidth links
sysctl -w net.core.rmem_max=26214400
sysctl -w net.core.wmem_max=26214400

# Enable TCP BBR congestion control for better throughput
sysctl -w net.ipv4.tcp_congestion_control=bbr

# Persist across reboots
cat <<EOF >> /etc/sysctl.d/99-vpn-performance.conf
net.core.rmem_max=26214400
net.core.wmem_max=26214400
net.ipv4.tcp_congestion_control=bbr
EOF

Expected behavior: VPN throughput improves by 15-30% on high-latency links. BBR congestion control adapts to packet loss better than the default Cubic algorithm.

Firewall Rules

# WireGuard
iptables -A INPUT -p udp --dport 51820 -j ACCEPT
iptables -A FORWARD -i wg0 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# OpenVPN
iptables -A INPUT -p udp --dport 1194 -j ACCEPT
iptables -A FORWARD -i tun0 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Expected behavior: VPN traffic flows through the firewall, forwarded to the internet with NAT.

Common Errors

1. Firewall blocking VPN port -- Forgot to open the VPN port in iptables or cloud security group. Clients connect but handshake never completes. Add the -A INPUT -p udp --dport rule for your chosen port.

2. Key mismatch -- Server and client key pairs do not correspond. WireGuard logs handshake did not complete. Regenerate both key pairs and redeploy the correct public keys.

3. MTU issues -- Default 1500 MTU causes fragmentation over some networks. Set MTU = 1420 in the WireGuard interface to avoid this. Symptoms include connections that hang or time out on larger transfers.

4. DNS leak -- VPN tunnel forwards traffic but DNS queries go through the ISP resolver. Force DNS with DNS = 1.1.1.1 in the client config and block port 53 on the physical interface.

5. Split tunnel misconfiguration -- AllowedIPs = 0.0.0.0/0 sends all traffic through VPN. If you intend only certain routes, specify only those subnets. For example, AllowedIPs = 10.0.0.0/8, 192.168.1.0/24 routes only private traffic.

6. OpenVPN TLS handshake failure -- Certificate verification fails. Verify the CA, client cert, and key match. Check verify-x509-name if used. Run openvpn --config client.ovpn --verb 4 for detailed TLS debugging.

Practice Questions

1. What is the difference between WireGuard and OpenVPN? WireGuard uses ChaCha20 encryption with a simpler 4,000-line codebase and kernel integration, while OpenVPN uses OpenSSL with AES-256-GCM and has a larger feature set including TCP fallback and certificate revocation.

2. Why should you set AllowedIPs to 0.0.0.0/0 in a full-tunnel VPN? It tells the operating system to route all IPv4 traffic through the VPN interface, preventing split-tunnel leaks where non-VPN traffic bypasses encryption.

3. What does iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE do? It enables NAT so traffic from VPN clients (10.x.x.x) appears to originate from the server's public IP when leaving the physical interface.

4. How does WireGuard handle roaming? WireGuard encrypts packets using the peer's public key regardless of source IP, so a client can change networks without reconnecting.

5. Challenge: Set up a WireGuard site-to-site tunnel between two cloud VPS instances in different regions. Configure iptables to forward traffic between the two private subnets.

Mini Project

Deploy a WireGuard VPN server on a $5/month VPS. Configure three client profiles for your phone, laptop, and a colleague. Verify with wg show that all peers have recent handshake timestamps, then test by visiting whatsmyip.com through the tunnel.

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.