Skip to content

Building Custom Kubernetes Controllers in Go: From CRD to Controller

DodaTech 6 min read

In this tutorial, you'll learn about Building Custom Kubernetes Controllers in Go: From CRD to Controller. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Custom Kubernetes controllers extend the Kubernetes API by watching Custom Resource Definitions and reconciling the desired state, enabling automation for domain-specific resources that the built-in controllers do not handle.

What You'll Learn

This tutorial covers designing Custom Resource Definitions, scaffolding a controller project with kubebuilder, implementing the reconcile loop, handling create-update-delete events, testing with envtest, and deploying the controller on a cluster.

Why It Matters

Built-in Kubernetes resources cover generic workloads but cannot model domain-specific concepts like database instances, certificate requests, or application releases. Custom controllers encapsulate operational knowledge into code that runs continuously, reducing manual intervention and human error.

Real-World Use

CoreOS built the etcd-operator using custom controllers to automate etcd cluster management -- backup, scaling, and failover. Kubernetes projects like cert-manager, Crossplane, and Knative are all custom controllers.

graph TD
  A[CRD: Database] --> B[Custom Controller]
  B --> C{Reconcile Loop}
  C --> D[Create Deployment]
  C --> E[Create Service]
  C --> F[Create PVC]
  D --> G[Database Pod Running]
  E --> H[Network Endpoint]
  F --> I[Persistent Storage]
  G --> J[Status Updated on CR]

Expected output: diagram showing how a custom controller watches a Database CRD and reconciles by creating Deployment, Service, and PVC resources.

Custom Resource Definition

Define a custom resource type with validation.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.example.com
spec:
  group: example.com
  names:
    kind: Database
    plural: databases
    singular: database
    shortNames:
    - db
  scope: Namespaced
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            required: ["engine", "version", "storage"]
            properties:
              engine:
                type: string
                enum: ["postgres", "mysql", "redis"]
              version:
                type: string
              storage:
                type: integer
                minimum: 1
              replicas:
                type: integer
                minimum: 1
                default: 3
              backup:
                type: object
                properties:
                  enabled:
                    type: boolean
                  schedule:
                    type: string

# Apply the CRD to the cluster
kubectl apply -f database-crd.yaml

# Verify the CRD is registered
kubectl get crd databases.example.com

# Create a custom resource instance
kubectl apply -f example-database.yaml

Expected output: the CRD shows ESTABLISHED status. Custom resources can be managed with kubectl get databases just like built-in resources.

Scaffolding a Controller with Kubebuilder

Kubebuilder generates the project structure for a controller.

# Install kubebuilder
curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
chmod +x kubebuilder && sudo mv kubebuilder /usr/local/bin/

# Scaffold the project
kubebuilder init --domain example.com --repo github.com/myorg/database-operator
kubebuilder create api --group example --version v1 --kind Database --resource --controller

Expected output: the project scaffold includes a main.go, controllers/database_controller.go, and api/v1/database_types.go ready for implementation.

The Reconcile Loop

The reconcile function runs whenever a watched resource changes.

package controllers

import (
	"context"
	"fmt"

	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/errors"
	"k8s.io/apimachinery/pkg/types"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/log"

	databasev1 "github.com/myorg/database-operator/api/v1"
)

type DatabaseReconciler struct {
	client.Client
}

func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := log.FromContext(ctx)

	var db databasev1.Database
	if err := r.Get(ctx, req.NamespacedName, &db); err != nil {
		if errors.IsNotFound(err) {
			return ctrl.Result{}, nil
		}
		return ctrl.Result{}, err
	}

	logger.Info("Reconciling Database", "engine", db.Spec.Engine, "version", db.Spec.Version)

	deploymentName := fmt.Sprintf("%s-%s", db.Name, db.Spec.Engine)
	var existingDeployment appsv1.Deployment
	err := r.Get(ctx, types.NamespacedName{Name: deploymentName, Namespace: db.Namespace}, &existingDeployment)

	if err != nil && errors.IsNotFound(err) {
		deployment := r.buildDeployment(&db, deploymentName)
		if err := r.Create(ctx, deployment); err != nil {
			return ctrl.Result{}, err
		}
		logger.Info("Created deployment", "name", deploymentName)
		return ctrl.Result{}, nil
	}

	if err != nil {
		return ctrl.Result{}, err
	}

	return ctrl.Result{}, nil
}

func (r *DatabaseReconciler) buildDeployment(db *databasev1.Database, name string) *appsv1.Deployment {
	replicas := db.Spec.Replicas
	return &appsv1.Deployment{
		ObjectMeta: metav1.ObjectMeta{
			Name:      name,
			Namespace: db.Namespace,
		},
		Spec: appsv1.DeploymentSpec{
			Replicas: &replicas,
			Selector: &metav1.LabelSelector{
				MatchLabels: map[string]string{"app": name},
			},
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{
					Labels: map[string]string{"app": name},
				},
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{
						{
							Name:  db.Spec.Engine,
							Image: fmt.Sprintf("%s:%s", db.Spec.Engine, db.Spec.Version),
						},
					},
				},
			},
		},
	}
}

// SetupWithManager registers the controller with the manager
func (r *DatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&databasev1.Database{}).
		Owns(&appsv1.Deployment{}).
		Complete(r)
}

Testing with Envtest

Run controller tests locally without a real cluster.

func TestDatabaseReconciler(t *testing.T) {
	env := &envtest.Environment{
		CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")},
	}

	cfg, err := env.Start()
	if err != nil {
		t.Fatal(err)
	}
	defer env.Stop()

	mgr, err := manager.New(cfg, manager.Options{})
	if err != nil {
		t.Fatal(err)
	}

	db := &databasev1.Database{
		ObjectMeta: metav1.ObjectMeta{
			Name:      "test-db",
			Namespace: "default",
		},
		Spec: databasev1.DatabaseSpec{
			Engine:   "postgres",
			Version:  "16",
			Storage:  10,
			Replicas: 3,
		},
	}

	c := mgr.GetClient()
	err = c.Create(context.Background(), db)
	if err != nil {
		t.Fatal(err)
	}

	var deployment appsv1.Deployment
	Eventually(func() error {
		return c.Get(context.Background(),
			types.NamespacedName{Name: "test-db-postgres", Namespace: "default"},
			&deployment)
	}, time.Second*10).Should(Succeed())
}

# Run controller tests
cd database-operator
go test ./controllers/ -v

# Expected output
# --- PASS: TestDatabaseReconciler (3.45s)
# PASS
# ok  	github.com/myorg/database-operator/controllers	3.456s

Expected output: the test passes after the controller creates the expected Deployment within 10 seconds.

Deploying the Controller

Build and deploy the controller to the cluster.

# Build the controller image
make docker-build IMG=myregistry/database-operator:v1.0.0

# Push to registry
make docker-push IMG=myregistry/database-operator:v1.0.0

# Deploy to cluster
make deploy IMG=myregistry/database-operator:v1.0.0

# Check controller logs
kubectl -n database-operator-system logs deployment/database-operator-controller-manager

Expected output: the controller manager pod starts and begins watching for Database custom resources. Logs show the controller starting and registering watches.

Practice Questions

  1. What triggers the reconcile function in a controller? Any create, update, or delete event on the watched custom resource or on owned resources (configured with Owns in SetupWithManager).

  2. How does the controller handle errors during reconciliation? The reconcile function returns an error and a requeue duration. The controller retries with exponential backoff. Permanent errors should be logged and not returned to avoid infinite retry loops.

  3. What is the purpose of the Finalizer in a custom controller? Finalizers prevent deletion of custom resources until the controller has cleaned up associated external resources (like cloud storage or DNS records). Without finalizers, the CR is deleted immediately, leaving orphaned resources.

Frequently Asked Questions

Do I need to write the controller in Go?

No. While Go is the most common choice because of the excellent client-go and controller-runtime libraries, you can write controllers in any language that can make HTTP requests to the Kubernetes API server. Python has the pykube-ng library, Java has the fabric8 client, and there are community-maintained SDKs for Rust and .NET.

How do I manage controller permissions?

Create a ClusterRole with only the permissions the controller needs (get, list, watch, create, update for its CRD and any owned resources). Bind it to the controller's service account. Use the lowest-privilege approach -- the controller should not have cluster-admin access.

What is the difference between a controller and an operator?

An operator is a controller that also manages the full lifecycle of the application it represents -- including installation, upgrades, backups, scaling, and failure recovery. All operators are controllers, but not all controllers are operators. An operator typically includes domain knowledge encoded into the reconciliation logic.

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.

← Previous Service Mesh: Istio vs Linkerd for Traffic Management & Security Next → Multi-Cluster Kubernetes: Management, Federation & Service Mesh

Built by the developers of DodaTech

Doda Browser, DodaZIP & Durga Antivirus Pro

Home Browse Kubernetes