Kubernetes Networking — CNI Plugins, Network Policies, and Service Mesh Integration

In this tutorial, you'll learn about Kubernetes Networking. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Kubernetes networking provides a flat network model where every Pod gets a unique IP address and can communicate with any other Pod without NAT, with CNI plugins implementing the underlying network fabric and Network Policies controlling traffic flow.

What You'll Learn

Why It Matters

Kubernetes networking is complex but fundamental to cluster security and application reliability. Without understanding CNI plugins, teams face mysterious connectivity issues. Without Network Policies, every Pod can talk to every other Pod, creating a massive security risk if one workload is compromised. A single vulnerable Pod in a flat network can scan and exploit every other application in the cluster.

Real-World Use

DodaTech runs Durga Antivirus Pro across multiple Kubernetes clusters with Calico CNI enforcing 150+ Network Policies, Cilium for eBPF-based Observability, and a service mesh for mTLS between Microservices. This layered networking approach contains breaches and provides full traffic visibility.

flowchart TD
    A["Pod A: 10.0.1.5"] -->|"CNI: Calico"| B["Node 1: eth0"]
    B -->|"Overlay/VXLAN"| C["Node 2: eth0"]
    C -->|"CNI: Calico"| D["Pod B: 10.0.2.7"]
    A -.->|"Blocked by NetworkPolicy"| E["Pod C: 10.0.3.9"]
    F["NetworkPolicy: default-deny"] -->|"Deny all ingress"| A
    G["NetworkPolicy: allow-api"] -->|"Allow port 8080 from frontend"| H["Pod: API v2"]
    I["CoreDNS"] --> J["Service DNS: my-svc.namespace.svc.cluster.local"]
    J --> K["Service -> Pod IPs"]
    style A fill:#326CE5,color:#fff
    style D fill:#326CE5,color:#fff
    style H fill:#326CE5,color:#fff

CNI Plugin Architecture

Container Network Interface plugins implement the Kubernetes networking model. The choice of CNI affects performance, security features, and operational complexity.

Network Policies

Network Policies are firewall rules for Pods. They are namespace-scoped and use label selectors to define allowed traffic. By default, all Pod-to-Pod traffic is allowed -- Network Policies change this to a deny-by-default model when applied.

# default-deny-all.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress

Expected behavior: No ingress traffic is allowed to any Pod in the production namespace. All existing connections are dropped. Any new connection attempt from another Pod is blocked unless a more specific NetworkPolicy allows it.

# allow-api-from-frontend.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-traffic
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

Expected behavior: Only Pods with label app: frontend can send TCP traffic to port 8080 on Pods with label app: api. All other ingress traffic to api Pods is blocked by the default-deny policy.

# allow-db-from-api-only.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-db-access
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: database
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: api
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - protocol: TCP
          port: 5432

Expected behavior: Database Pods accept traffic only from Pods with label app: api in the same namespace and from any Pod in the monitoring namespace (for monitoring tools). This prevents the frontend from directly accessing the database.

Egress Network Policies

Control outbound traffic from Pods to prevent data exfiltration and restrict access to external services.

# restrict-egress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-egress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: database
      ports:
        - protocol: TCP
          port: 5432
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8
              - 172.16.0.0/12
              - 192.168.0.0/16
      ports:
        - protocol: TCP
          port: 443

Expected behavior: The API Pods can only connect to database Pods on port 5432 and to external HTTPS endpoints (port 443) on the internet. All traffic to private IP ranges (RFC 1918) except the database is blocked, preventing the API from reaching other internal services if compromised.

Cilium and eBPF for Advanced Networking

Cilium uses eBPF to implement networking, security, and Observability at the kernel level without modifying the application or sidecar proxies.

# cilium-network-policy.yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-layer7-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: api
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/api/v2/public/*"
              - method: "POST"
                path: "/api/v2/auth/*"

Expected behavior: Cilium inspects HTTP requests at L7 and allows only GET requests to public endpoints and POST requests to auth endpoints. Any other HTTP method or path is blocked, providing application-layer security beyond what standard Network Policies (L3/L4 only) can achieve.

# Verify Cilium is working
kubectl -n kube-system exec daemonset/cilium -- cilium status

# Expected output:
# KVStore:                Ok   Disabled
# Kubernetes:             Ok   1.0 (v1.28.0) [linux/amd64]
# Kubernetes APIs:        ["Endpoint", "CiliumNode", "CiliumNetworkPolicy", ...]
# Cilium:                 Ok   OK
# NodeMonitor:            Listening for events on 4 CPUs with 64x4096 shared memory
# Cilium health daemon:   Ok

# Test connectivity
kubectl run test-pod --image=busybox -it --rm -- wget -qO- http://api-service:8080/api/v2/public/healthz

# Expected output (if policy allows):
# {"status":"ok"}

Service Mesh Integration

Service meshes like Istio and Linkerd add mTLS, traffic splitting, and Observability on top of Kubernetes networking.

# istio-peerauthentication.yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: strict-mtls
  namespace: production
spec:
  mtls:
    mode: STRICT

Expected behavior: All traffic between services in the production namespace is encrypted with mutual TLS. Communication is authenticated and encrypted at the application layer, adding defense in depth beyond Network Policies.

Common Errors

1. Network Policies not working because no CNI plugin supports them: Flannel does not enforce Network Policies. If you apply a NetworkPolicy with Flannel, it has no effect. Always verify your CNI plugin supports policy enforcement (Calico, Cilium, Weave, Antrea).

2. Forgetting podSelector in NetworkPolicy: A NetworkPolicy with an empty podSelector ({}) affects all Pods in the namespace. This is useful for default-deny policies but dangerous if applied by mistake to a production namespace without understanding the impact.

3. CIDR exceptions in egress policies not covering all private ranges: The standard RFC 1918 ranges are 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16. Missing any of these in the except block allows traffic to those private IPs, potentially bypassing your security controls.

4. DNS resolution failures when egress policies block port 53: CoreDNS runs inside the cluster. If an egress policy blocks UDP port 53, Pods cannot resolve DNS names. Always include an explicit egress rule for DNS: kube-system namespace on port 53/UDP.

5. Overlapping Network Policies causing unexpected allow rules: Network Policies are additive. If one policy allows traffic from app: frontend on port 8080 and another allows all traffic from app: monitoring with no port restriction, the effective rule is the union. Traffic from monitoring is allowed on all ports. Use a default-deny policy to start from a clean slate.

Practice Questions

1. What is the default networking behavior in Kubernetes when no Network Policies exist? Answer: By default, all Pod-to-Pod traffic is allowed across all namespaces. There is no isolation. This flat network model is convenient for development but dangerous for production with multi-tenant or sensitive workloads.

2. How does a NetworkPolicy with podSelector: {} differ from one with podSelector: matchLabels: app: api? Answer: An empty podSelector ({}) matches all Pods in the namespace. A labeled selector matches only Pods with that label. The empty selector is used for default-deny policies.

3. Why does Calico support Network Policies but Flannel does not? Answer: Flannel only handles IP address management and packet Encapsulation (L2/L3). It does not implement a firewall or policy engine. Calico includes a distributed firewall that enforces policy rules at each node using iptables or eBPF.

4. What is the difference between a Kubernetes NetworkPolicy and a CiliumNetworkPolicy? Answer: Kubernetes NetworkPolicy operates at L3/L4 (IP addresses, ports, protocols). CiliumNetworkPolicy extends this to L7 (HTTP methods, paths, API endpoints) and supports DNS-based rules, CIDR rules with FQDN resolution, and Kafka/gRPC-aware policies.

Challenge

Design a zero-trust network for a three-tier application (frontend, API, database) with these requirements: all namespaces default to deny-all ingress and egress, frontend can only reach API on port 8080, API can only reach database on port 5432, database has no egress access, monitoring namespace (Prometheus) can scrape metrics on all tiers, API can reach external HTTPS services on port 443 only, and DNS resolution is allowed for all Pods. Implement with both standard Network Policies and one Cilium L7 policy that restricts API access to specific HTTP paths.

Mini Project

Build a complete network isolation framework for a Microservices platform: install Calico or Cilium on a kind cluster, create namespaces (frontend, backend, data, monitoring), deploy sample applications in each namespace, implement namespace-level default-deny policies, create tier-specific ingress policies (frontend->backend->data chain), configure egress policies that restrict outbound traffic to only necessary external endpoints, set up a Cilium Hubble UI for traffic visualization, generate traffic with a load testing tool, and observe blocked/allowed flows in Hubble. Document the policy hierarchy and test each rule with connectivity probes.

Related Resources

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.