Kubernetes HPA and VPA Autoscaling — Complete Guide with Metrics Server and Custom Metrics

In this tutorial, you'll learn about Kubernetes HPA and VPA Autoscaling. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Kubernetes autoscaling automatically adjusts the number of Pod replicas or their resource allocations based on observed metrics, ensuring applications handle traffic spikes without manual intervention while minimizing cost during low demand.

What You'll Learn

Why It Matters

Hard-coding replica counts and resource requests leads to either over-provisioning (wasting money) or under-provisioning (causing outages during traffic spikes). The Horizontal Pod Autoscaler (HPA) adjusts replica counts based on CPU, memory, or custom metrics. The Vertical Pod Autoscaler (VPA) adjusts CPU and memory requests and limits to right-size containers. Together they eliminate manual scaling decisions and reduce infrastructure costs.

Real-World Use

DodaTech runs Durga Antivirus Pro's scan workers with an HPA that scales from 5 to 50 replicas based on queue depth from Kafka, while the VPA right-sizes memory requests for the API layer to avoid wasted resources during low-traffic periods.

flowchart TD
    A["Metrics Server / Prometheus"] --> B["HPA Controller"]
    C["Kubernetes API"] --> B
    B --> D["Scale: Deployment/ReplicaSet"]
    D --> E["Desired Replicas"]
    A --> F["VPA Recommender"]
    C --> F
    F --> G["VPA Updater"]
    G --> H["Updated Pod Resources"]
    H --> I["Evict + Recreate Pod"]
    E --> J["Traffic Spike -> Scale Up"]
    E --> K["Traffic Low -> Scale Down"]
    style B fill:#326CE5,color:#fff
    style F fill:#326CE5,color:#fff

Metrics Server Setup

The HPA requires resource metrics (CPU, memory) to be available. Metrics Server collects these from the kubelet.

# Install Metrics Server
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Expected output:
# serviceaccount/metrics-server created
# clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
# clusterrole.rbac.authorization.k8s.io/metrics-server:aggregated-metrics-reader created
# rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
# ...

# Verify metrics are available
kubectl top nodes

# Expected output:
# NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
# node-01    450m         22%    2048Mi           26%
# node-02    380m         19%    1890Mi           24%

Horizontal Pod Autoscaler

The HPA scales replicas based on observed metrics. It periodically queries the metrics endpoint and calculates the desired replica count.

# hpa-cpu.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-deployment
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Expected behavior: When the average CPU utilization across all Pods exceeds 70%, the HPA increases replicas up to 20. When utilization drops below 70%, it scales down to a minimum of 3.

# Create the HPA
kubectl apply -f hpa-cpu.yaml

# Expected output:
# horizontalpodautoscaler.autoscaling/api-hpa created

# Generate traffic to trigger scaling
kubectl run -it load-generator --image=busybox -- sh -c "while true; do wget -q -O- http://api-service; done"

# Watch the HPA
kubectl get hpa api-hpa -w

# Expected output (shows scaling up and down):
# NAME      REFERENCE              TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
# api-hpa   Deployment/api-deploy  45%/70%   3         20        3          1m
# api-hpa   Deployment/api-deploy  85%/70%   3         20        5          2m
# api-hpa   Deployment/api-deploy  120%/70%  3         20        8          3m
# api-hpa   Deployment/api-deploy  65%/70%   3         20        8          5m

Multi-Metric HPA

Combine CPU, memory, and custom metrics for more intelligent scaling decisions.

# hpa-multi.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: worker-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: worker-deployment
  minReplicas: 2
  maxReplicas: 50
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 70
    - type: Pods
      pods:
        metric:
          name: queue_depth
        target:
          type: AverageValue
          averageValue: 10
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 1
          periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Pods
          value: 4
          periodSeconds: 15

Expected behavior: The HPA uses the metric that results in the highest number of replicas. The behavior block configures more aggressive scaling up (add 4 Pods every 15 seconds) and conservative scaling down (remove 1 Pod every 60 seconds, with a 5-minute stabilization window). The queue_depth custom metric requires a custom metrics adapter like Prometheus Adapter.

Vertical Pod Autoscaler

The VPA recommends and automatically adjusts CPU and memory requests. It operates in three modes: Off (recommendations only), Auto (evict and recreate Pods with new resources), and Initial (only applies to new Pods).

# vpa.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-deployment
  updatePolicy:
    updateMode: Auto
  resourcePolicy:
    containerPolicies:
      - containerName: "*"
        minAllowed:
          cpu: 100m
          memory: 128Mi
        maxAllowed:
          cpu: 2000m
          memory: 2Gi
        controlledResources: ["cpu", "memory"]

Expected behavior: The VPA recommender analyzes historical resource usage and sets new CPU/memory requests. In Auto mode, the VPA Updater evicts Pods whose resource requests deviate significantly from the recommendation. The Deployment controller recreates the Pod with updated resources.

# Install VPA
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler
./hack/vpa-up.sh

# Expected output:
# customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalers.autoscaling.k8s.io created
# ...

# View VPA recommendations
kubectl describe vpa api-vpa

# Expected output (abbreviated):
# Status:
#   Recommendation:
#     Container Recommendations:
#       Container Name:  api
#       Lower Bound:
#         Cpu:     150m
#         Memory:  256Mi
#       Target:
#         Cpu:     250m
#         Memory:  384Mi
#       Upper Bound:
#         Cpu:     800m
#         Memory:  1Gi

HPA vs VPA Decision Matrix

Common Errors

1. Metrics Server not installed: The HPA cannot calculate target metrics without Metrics Server. kubectl get hpa shows <unknown> targets. Always verify kubectl top pods works before creating HPAs.

2. Missing resource requests on containers: The HPA uses the requests value as the baseline for utilization calculations. If a container has no requests.cpu, the HPA cannot calculate utilization and ignores that metric.

3. VPA and HPA on the same Deployment without careful configuration: VPA Auto mode changes resource requests, which affects HPA utilization calculations. This can cause oscillation. The recommended pattern is to use HPA for stateless workloads and VPA for stateful ones, or use VPA in Off mode for recommendations only.

4. Stabilization window too short for scale-down: Without a stabilization window, the HPA scales down immediately when a traffic spike ends, then scales up again when the next request comes. This causes thrashing. Set stabilizationWindowSeconds to at least 300 seconds for scale-down.

5. Custom metrics Adapter not configured for application metrics: The HPA can use custom metrics only if a metrics adapter (Prometheus Adapter, Datadog Cluster Agent) exposes them through the Kubernetes custom metrics API. Without the Adapter, type: Pods metrics fail with an error.

Practice Questions

1. How does the HPA calculate the desired number of replicas? Answer: The HPA divides the current metric value by the target value and multiplies by the current replica count. If current CPU is 80% and target is 70% with 5 replicas, the result is ceil(80/70 * 5) = 6 replicas. When multiple metrics are used, the metric that produces the highest replica count is applied.

2. What happens when the VPA and HPA target the same Deployment? Answer: The VPA changes resource requests while the HPA changes replica counts. If the VPA increases CPU requests, the HPA sees lower utilization and scales down, potentially causing the VPA to recommend even higher requests. This feedback loop can cause instability. Use VPA in Off mode for recommendations with HPA, or target different Deployments.

3. What is the difference between HPA scale-up and scale-down policies? Answer: Scale-up policies are more aggressive by default (add Pods quickly to handle spikes), while scale-down policies are conservative (remove Pods slowly with a stabilization window to avoid flapping). These can be customized in the behavior field.

4. How does the VPA recommender calculate target resource requests? Answer: The recommender analyzes historical resource usage over the past 8 days (by default), computes a safe percentile (usually the 95th percentile), and adds a safety margin. The target is the resource amount needed to handle peak load without wasting excess capacity.

Challenge

Deploy three Microservices with different scaling requirements: a web API that scales on CPU (target: 60%, min 2, max 15), a worker that scales on a custom metric jobs_pending (target: 5, min 1, max 30), and a database that uses VPA in Auto mode. Configure aggressive scale-up (add 4 Pods per 10 seconds) and conservative scale-down (remove 1 Pod per 5 minutes) for the API. Install Prometheus Adapter to expose the custom metric. Generate load and verify all three scaling behaviors.

Mini Project

Build a complete autoscaling demo for an e-commerce platform: deploy a Go API with requests: cpu: 200m, memory: 256Mi, configure an HPA with CPU (target 70%) and memory (target 80%), install the VPA in Off mode to collect recommendations for 24 hours, deploy a load generator that simulates Black Friday traffic with spikes from 100 to 5000 requests per second, monitor the HPA scaling behavior with kubectl get hpa -w, log the recommendation from VPA, and generate a report showing the cost savings of autoscaling compared to fixed 20-replica deployment over a 7-day period.

Related Resources

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.