Secure Deployment: CI/CD Security and Secure Release Management

In this tutorial, you will learn about Secure Deployment: CI/CD Security and Secure Release Management. We cover key concepts, practical examples, and best practices to help you master this topic.

Secure deployment ensures that the path from source code to production is protected against tampering, secrets leakage, and misconfigurations. It covers CI/CD pipeline hardening, artifact integrity, deployment strategies, and post-deployment verification.

flowchart LR
    Code[Source Code] -->|Signed Commit| Repo[Git Repository]
    Repo -->|Trigger| CIBuild[CI Build]
    CIBuild -->|Security Scan| Scan[SAST + Dependency Scan]
    Scan -->|Artifact| Build[Build Container Image]
    Build -->|Sign Image| Registry[Container Registry]
    Registry -->|Verify Signature| CD[CD Pipeline]
    CD -->|Canary| Canary[Canary Deploy 5%]
    CD -->|Staging| Staging[Staging Deploy]
    Staging -->|Integration Tests| Tests[Security Tests]
    Tests -->|Promote| Production[Production Deploy]

What You'll Learn

• CI/CD pipeline security: secrets, dependencies, artifact integrity
• Container image signing and verification (cosign)
• Immutable deployments and rollback
• Canary releases with automatic rollback

Why It Matters

A compromised CI/CD pipeline can inject malicious code into production undetected. The SolarWinds attack exploited a compromised build pipeline to distribute malware through trusted software updates.

Real-World Use

A DevOps team implements: signed commits with GPG, CI pipeline that scans for secrets before building, cosign-signed container images, and canary deployments that automatically roll back if error rate increases. A compromised dependency was caught by the CI pipeline before reaching production.

Secure Deployment Implementation

CI/CD Secrets Scanning

# .github/workflows/secret-scan.yml
name: Secret Scan
on: [pull_request, push]
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      # Scan for secrets in code
      - name: Scan for secrets
        uses: trufflesecurity/trufflehog@v3
        with:
          extra_args: --results=verified,unknown
      
      # Scan for secrets in git history
      - name: GitLeaks
        uses: gitleaks/gitleaks-action@v2
      
      # Prevent committing secrets
      - name: Check .env files
        run: |
          if git diff --cached --name-only | grep -q '\.env$'; then
            echo "Error: .env files should not be committed"
            exit 1
          fi

Expected output:

TruffleHog: No secrets found.
GitLeaks: No leaks detected in git history.

Container Image Signing with Cosign

#!/bin/bash
# CI/CD pipeline step: sign container image

# Generate key pair (store private key securely)
cosign generate-key-pair

# Build image
docker build -t myapp:${GIT_SHA} .
docker tag myapp:${GIT_SHA} registry.example.com/myapp:${GIT_SHA}

# Sign the image
COSIGN_PASSWORD=${COSIGN_KEY_PASSWORD} cosign sign \
  --key cosign.key \
  registry.example.com/myapp:${GIT_SHA}

# Verify before deployment
cosign verify \
  --key cosign.pub \
  registry.example.com/myapp:${GIT_SHA}

# Output: Verification OK

Expected output:

Image signed with cosign. Signature stored in registry alongside the image.
Verification confirms image was signed by the trusted CI pipeline.

Immutable Deployment with Rollback

class ImmutableDeployer {
  constructor(k8sClient) {
    this.k8s = k8sClient;
  }

  async deploy(serviceName, imageTag, config) {
    const deploymentName = `${serviceName}-${imageTag.replace(/[^a-zA-Z0-9-]/g, '-')}`;

    // Create new deployment (immutable: never update, always create new)
    await this.k8s.createDeployment({
      metadata: { name: deploymentName, labels: { app: serviceName, version: imageTag } },
      spec: {
        replicas: config.canaryPercent ? 1 : config.replicas,
        selector: { matchLabels: { app: serviceName, version: imageTag } },
        template: {
          metadata: { labels: { app: serviceName, version: imageTag } },
          spec: {
            containers: [{
              name: serviceName,
              image: `registry.example.com/${serviceName}:${imageTag}`,
              env: config.env,
              resources: config.resources,
              readinessProbe: config.readinessProbe,
              livenessProbe: config.livenessProbe
            }]
          }
        }
      }
    });

    // Wait for deployment to be ready
    await this.waitForReady(deploymentName, config.timeout || 300);

    return deploymentName;
  }

  async canaryDeploy(serviceName, imageTag, config) {
    // Deploy canary with 5% traffic
    const canaryDeployment = await this.deploy(serviceName, imageTag, {
      ...config,
      canaryPercent: 5,
      replicas: 1
    });

    // Monitor for canary period
    const healthy = await this.monitorCanary(canaryDeployment, config.canaryPeriod || 300);

    if (!healthy) {
      await this.rollback(canaryDeployment);
      return { success: false, reason: 'Canary health check failed' };
    }

    // Scale up canary to full capacity
    await this.scaleDeployment(canaryDeployment, config.replicas);

    // Update service selector
    await this.updateServiceSelector(serviceName, imageTag);

    return { success: true, deployment: canaryDeployment };
  }

  async rollback(deploymentName) {
    await this.k8s.deleteDeployment(deploymentName);
    // Traffic goes to previous deployment automatically
  }
}

Expected output:

Canary deployed with 5% traffic → monitored for 5 minutes → healthy → scaled to 100% → service updated.
If error rate increases during canary → automatic rollback.

Common Mistakes

• Storing secrets in CI/CD environment variables — use vault or secrets manager integration.
• Building images without version tags or using latest — impossible to track which version is deployed.
• Not signing artifacts — unsigned images can be replaced with malicious versions in the registry.
• Deploying directly to production without staging verification.
• Not having a rollback plan — every deployment should have a tested rollback procedure.

Practice Questions

1. Why is secret scanning important in CI/CD pipelines?
2. How does container image signing prevent tampering?
3. What is an immutable deployment?
4. How does a canary deployment reduce risk?
5. What is the difference between blue-green and canary deployment?

Challenge

Set up a secure CI/CD pipeline for a Node.js application. Implement: (1) secret scanning in PRs, (2) SAST on every commit, (3) container image signing with cosign, (4) canary deployment with automatic rollback on health check failure, (5) staging environment with security tests before production.

FAQ

Mini Project

Create a secure deployment pipeline for a Node.js API. Use GitHub Actions with: secret scanning (trufflehog), SAST (ESLint security), container build and sign (cosign), deploy to Kubernetes with canary Strategy (5% → 30 min monitoring → 100%), and automatic rollback on failure.

What's Next

Continue to Security Architecture to learn about designing secure backend architectures.