Kubernetes Secret Not Found Fix

In this tutorial, you'll learn about Kubernetes Secret Not Found Fix. We cover key concepts, practical examples, and best practices.

The Problem

Your pod fails to start with a Secret-related error:

Events:
  Type     Reason       Age   From               Message
  ----     ------       ----  ----               -------
  Warning  FailedMount  10s   kubelet            Secret "db-credentials" not found

Or the application logs show authentication failures because environment variables from the secret are empty or missing.

Quick Fix

Step 1: Verify the secret exists

kubectl get secret db-credentials

Error from server (NotFound): secrets "db-credentials" not found

If the secret does not exist, create it:

kubectl create secret generic db-credentials \
  --from-literal=DB_USER=admin \
  --from-literal=DB_PASSWORD=s3cret

Or from a file:

kubectl create secret generic db-credentials \
  --from-file=./credentials.json

Step 2: Check the namespace

Secrets are namespaced. A pod in namespace staging cannot see a secret in production:

kubectl get secret --all-namespaces | grep db-credentials

WRONG -- pod in staging references a secret in production:

apiVersion: v1
kind: Pod
metadata:
  namespace: staging
spec:
  containers:
  - env:
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: prod-db-credentials   # secret exists only in "production"
          key: password

RIGHT -- create the secret in the same namespace or use a sync tool.

Step 3: Verify secret key names

kubectl get secret db-credentials -o jsonpath='{.data}'

{"DB_PASSWORD":"czNjcjN0","DB_USER":"YWRtaW4="}

WRONG -- referencing a key that does not exist:

env:
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: db-credentials
      key: password   # actual key is "DB_PASSWORD"

RIGHT -- use the exact key name:

env:
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: db-credentials
      key: DB_PASSWORD

Step 4: Check if the secret is in kubernetes.io/basic-auth format

Secrets with type kubernetes.io/basic-auth must use username and password keys:

kubectl get secret db-credentials -o yaml

WRONG -- custom keys in basic-auth secret:

apiVersion: v1
kind: Secret
type: kubernetes.io/basic-auth
data:
  DB_USER: YWRtaW4=       # basic-auth requires "username" key
  DB_PASSWORD: czNjcjN0   # basic-auth requires "password" key

RIGHT -- use Opaque type for custom keys:

apiVersion: v1
kind: Secret
type: Opaque
data:
  DB_USER: YWRtaW4=
  DB_PASSWORD: czNjcjN0

Step 5: Check if the pod can access the secret via RBAC

Some environments (e.g., OpenShift) require explicit RBAC permissions to read secrets:

kubectl auth can-i get secret/db-credentials --as system:serviceaccount:default:my-sa

If no, bind a role with get permission on secrets to the service account.

Step 6: Check for immutable secrets

If the secret is immutable, it cannot be updated:

kubectl get secret db-credentials -o jsonpath='{.immutable}'

If true, you must create a new secret and update the deployment to reference the new name.

Use DodaTech's Secret Inspector to audit secret keys, verify pod-to-secret references, and detect missing or misconfigured secrets across namespaces.

Prevention

• Create secrets before the deployments that reference them.
• Keep secrets in the same namespace as their consuming pods.
• Use kubectl create secret --dry-run=client -o yaml to validate before applying.
• Use external secret operators (External Secrets Operator, Sealed Secrets) for GitOps workflows.
• Never log secret values or commit them to Git.

Common Mistakes with secret not found

1. Using foldl instead of foldl' causing stack overflow on large lists
2. Forgetting deriving (Show, Eq) on custom data types needed for debugging
3. Placing the wildcard pattern first in case expressions, making all subsequent patterns unreachable

These mistakes appear frequently in real-world K8S code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ