Jenkins Node/Agent Offline Fix

In this tutorial, you'll learn about Jenkins Node/Agent Offline Fix. We cover key concepts, practical examples, and best practices.

Your Jenkins jobs are stuck in the queue with Pending — master is offline or Node is not available — the agent disconnected or Jenkins marked it offline due to connection issues.

The Problem

Agent abc-agent is offline
Jenkins will wait until the agent is available to proceed with this job.
Builds waiting in queue: 15

The agent went offline because of a network hiccup, disk space exhaustion, or the agent process crashed. Jenkins holds all jobs targeting that agent until it reconnects.

Step-by-Step Fix

1. Check the agent log

SSH into the agent machine:

journalctl -u jenkins-agent -n 50
# or
cat /var/log/jenkins/jenkins-agent.log | tail -50

Look for java.lang.OutOfMemoryError, Disk full, or Connection refused errors.

2. Restart the agent

From Jenkins UI: Dashboard > Nodes > [Agent Name] > "Launch agent" button.

From the agent machine:

sudo systemctl restart jenkins-agent

3. Free up disk space

# Check disk usage
df -h

# Clean Jenkins workspace
rm -rf /var/lib/jenkins/workspace/*

# Clean old builds
# Jenkins UI: [Job] > Build History > "Delete Builds" (older than 30 days)

4. Configure agent auto-restart

# /etc/systemd/system/jenkins-agent.service
[Service]
ExecStart=/usr/bin/java -jar /opt/jenkins/agent.jar -jnlpUrl https://jenkins.example.com/computer/agent-name/jenkins-agent.jnlp
Restart=always
RestartSec=10
User=jenkins

5. Increase agent memory

# In agent startup command or systemd config
ExecStart=/usr/bin/java -Xmx2g -jar /opt/jenkins/agent.jar ...

6. Set up monitoring

// Jenkins pipeline to check agent health
pipeline {
    agent { label 'linux' }
    stages {
        stage('Health Check') {
            steps {
                sh '''
                    echo "Disk: $(df -h / | tail -1)"
                    echo "Memory: $(free -h | grep Mem)"
                    echo "Load: $(uptime)"
                '''
            }
        }
    }
}

Expected output:

Agent abc-agent reconnected
Disk: /dev/sda1   50G   20G   30G   40% /
Memory: 7.6G total, 2.1G used
Agent status: Online

Prevention Tips

• Set Restart=always in the agent systemd service
• Monitor disk space and memory on agent machines
• Use ephemeral agents (Docker, Kubernetes) for self-healing
• Configure -Xmx based on available memory
• Set up agent health checks with automatic replacement

Common Mistakes with agent offline

1. Forgetting that lazy evaluation defers computation until the value is forced, causing space leaks with unevaluated thunks
2. Using return to exit a function early instead of wrapping a pure value in the monad
3. Mixing let bindings with <- bindings in do notation, producing type errors

These mistakes appear frequently in real-world JENKINS code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ