Prometheus Kubernetes

In this tutorial, you'll learn about Prometheus on Kubernetes: kube. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

The kube-Prometheus-stack is a Helm chart that deploys Prometheus, Alertmanager, Grafana, and the Prometheus Operator on Kubernetes, providing a complete monitoring solution with preconfigured dashboards and alerting rules for cluster infrastructure.

What You'll Learn

In this tutorial, you will deploy kube-Prometheus-stack on a Kubernetes cluster using Helm, explore the preconfigured dashboards and alerts, and customize the deployment for your workloads.

Why It Matters

Kubernetes clusters are complex Distributed Systems with many components: control plane, nodes, pods, services, and persistent volumes. Monitoring them manually is infeasible. The kube-Prometheus-stack gives you a production-ready monitoring platform in a single Helm install, with dashboards for every Kubernetes component and alerts for common failure modes.

Real-World Use

DodaZIP runs its compression workers on Kubernetes. The kube-Prometheus-stack monitors cluster health, pod resource usage, and worker queue depth. Preconfigured alerts notify the team when pods are in CrashLoopBackOff, when nodes are under memory pressure, or when worker pods cannot keep up with the job queue.

Step 1: Add the Helm Repository

Add the Prometheus community Helm repository:

helm repo add <a href="/devops/prometheus-grafana/">Prometheus</a>-community https://prometheus-community.github.io/helm-charts
helm repo update

Expected output:

"<a href="/devops/prometheus-grafana/">Prometheus</a>-community" has been added to your repositories

Step 2: Install kube-Prometheus-stack

Create a namespace and install the stack:

kubectl create namespace monitoring
helm install <a href="/devops/prometheus-grafana/">Prometheus</a>-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring

Expected output:

NAME: <a href="/devops/prometheus-grafana/">Prometheus</a>-stack
LAST DEPLOYED: ...
NAMESPACE: monitoring
STATUS: deployed

Step 3: Verify the Deployment

Check that all components are running:

kubectl get pods -n monitoring

Expected output:

NAME                                                     READY   STATUS
alertmanager-prometheus-stack-kube-prome-alertmanager-0   2/2     Running
prometheus-prometheus-stack-kube-prome-prometheus-0       2/2     Running
prometheus-stack-grafana-7d9b8f5c9b-abcde                 3/3     Running
prometheus-stack-kube-state-metrics-6f5d7c8d9-x1y2z       1/1     Running
<a href="/devops/prometheus-grafana/">Prometheus</a>-stack-<a href="/devops/prometheus-grafana/">Prometheus</a>-node-exporter-abcde            1/1     Running

Step 4: Access Grafana

Port-forward to the Grafana service:

kubectl port-forward service/<a href="/devops/prometheus-grafana/">Prometheus</a>-stack-grafana 3000:80 -n monitoring

Open http://localhost:3000. Default login is admin / prom-operator. The stack includes preconfigured dashboards for Kubernetes:

• Kubernetes / Compute Resources / Cluster
• Kubernetes / Compute Resources / Namespace (Pods)
• Kubernetes / Compute Resources / Node (Pods)
• Kubernetes / Networking
• Kubernetes / Persistent Volumes

Step 5: View Alerting Rules

The stack includes built-in alerting rules for Kubernetes. View them in Prometheus at /rules or in Alertmanager:

kubectl port-forward service/<a href="/devops/prometheus-grafana/">Prometheus</a>-stack-kube-prome-<a href="/devops/prometheus-grafana/">Prometheus</a> 9090:9090 -n monitoring

Preconfigured alerts include KubePodCrashLooping, KubeJobFailed, KubeNodeUnreachable, KubeCPUOvercommit, and KubeMemoryOvercommit.

Customizing the Stack

Override default values with a custom values file:

prometheus:
  prometheusSpec:
    retention: 30d
    resources:
      requests:
        memory: 8Gi

grafana:
  adminPassword: my-secure-password
  ingress:
    enabled: true
    hosts:
      - grafana.example.com

Upgrade the release with custom values:

helm upgrade <a href="/devops/prometheus-grafana/">Prometheus</a>-stack <a href="/devops/prometheus-grafana/">Prometheus</a>-community/kube-<a href="/devops/prometheus-grafana/">Prometheus</a>-stack \
  -f custom-values.yaml -n monitoring

Common Mistakes

1. Insufficient Resource Allocation

The full stack requires significant resources. Set resource requests and limits for each component, especially Prometheus and Grafana, to avoid OOM kills.

2. Exposing Services Without TLS

Never expose Prometheus or Alertmanager directly without authentication. Use ingress with TLS and basic auth or OAuth.

3. Not Configuring Persistent Storage

By default, Prometheus uses emptyDir for storage. When the pod restarts, metrics data is lost. Configure persistent volume claims for production.

4. Using Default Grafana Credentials

Change the default admin password immediately. Use environment variables or Kubernetes secrets for secure configuration.

5. Ignoring Pod Security Policies

The stack creates privileged containers (Node Exporter). Ensure your cluster allows privileged containers or configure security context constraints.

Practice Questions

1. What components does kube-Prometheus-stack deploy? Prometheus, Alertmanager, Grafana, Prometheus Operator, kube-state-metrics, and Node Exporter.

2. How do you access preconfigured Grafana dashboards? Port-forward to the Grafana service and log in. The Kubernetes dashboards are available in the Grafana Dashboards menu.

3. What are some built-in alerts that come with the stack? KubePodCrashLooping, KubeJobFailed, KubeNodeUnreachable, KubeCPUOvercommit, KubeMemoryOvercommit.

4. How do you customize Prometheus retention in the stack? Set <a href="/devops/prometheus-grafana/">Prometheus</a>.prometheusSpec.retention in a custom values file and upgrade the Helm release.

5. Challenge: Deploy kube-Prometheus-stack with persistent storage for Prometheus, a custom retention of 45 days, and a secured Grafana with TLS ingress.

What's Next

Take Prometheus to production with retention tuning, high availability, and Thanos for long-term storage and global querying.

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.