Hadoop NameNode Safe Mode Fix

In this tutorial, you'll learn about Hadoop NameNode Safe Mode Fix. We cover key concepts, practical examples, and best practices.

Running HDFS commands produces:

mkdir: Cannot create directory /data. Name node is in safe mode.

The NameNode enters safe mode when the block replication threshold is not met. HDFS stays in safe mode until a configurable percentage of blocks are reported as healthy by DataNodes. This happens after a cluster restart, DataNode failure, or disk issues.

Step-by-Step Fix

1. Check safe mode status

WRONG — guessing whether the cluster is in safe mode:

RIGHT — check status:

hdfs dfsadmin -safemode get
# Safe mode is ON

The output shows "Safe mode is ON" or "Safe mode is OFF." Also check the block report:

hdfs dfsadmin -report
# Look for: "Safe mode is ON"
# "Live datanodes: 3" (should match expected count)

2. Wait for automatic exit

WRONG — rushing to force exit:

# Safe mode exits automatically when:
# - DataNodes report back >99.9% of blocks
# - The safemode threshold is met

RIGHT — monitor the progress:

# Check every 30 seconds
watch -n 30 "hdfs dfsadmin -safemode get; hdfs dfsadmin -report | grep 'Configured Capacity'"

If the cluster has enough healthy DataNodes, safe mode exits automatically within minutes.

3. Force exit safe mode (only if all DataNodes are healthy)

WRONG — forcing exit without checking DataNode health:

RIGHT — verify health first:

# Check that all nodes are live
hdfs dfsadmin -report | grep -E "Live|Dead|Decommission"

# If all expected DataNodes are live, force exit:
hdfs dfsadmin -safemode leave

Expected output: Safe mode is OFF.

4. Adjust safe mode threshold

WRONG — threshold too high for a small cluster:

# Default threshold is 0.999 (99.9% of blocks must be replicated)

RIGHT — lower the threshold for development:

hdfs dfsadmin -safemode threshold 0.95

Or permanently in hdfs-site.xml:

<property>
    <name>dfs.namenode.safemode.threshold-pct</name>
    <value>0.95f</value>
</property>

5. Fix under-replicated blocks

WRONG — blocks stuck at under-replicated count:

hdfs fsck / | grep -E "Under replicated|Missing blocks"

RIGHT — increase replication or fix DataNodes:

# Increase replication factor for critical data
hdfs dfs -setrep -R 3 /data

# Trigger block replication
hdfs dfsadmin -triggerBlockReport localhost:50010

# If blocks are missing, recover from trash or re-upload

6. Check DataNode connectivity

# List all DataNodes
hdfs dfsadmin -report

# Check connectivity to each DataNode
for node in $(hdfs dfsadmin -report | grep "Name:" | awk '{print $2}'); do
    ping -c 1 $node
done

If DataNodes are missing, start them:

hdfs datanode
# Or via service manager:
sudo systemctl start hadoop-datanode

Expected output: HDFS exits safe mode and file operations succeed.

Prevention

• Monitor DataNode health and block replication status.
• Set realistic safe mode thresholds for your cluster size.
• Ensure all DataNodes start before the NameNode times out.
• Configure dfs.namenode.safemode.extension to give DataNodes more time.
• Alert on persistent safe mode via monitoring systems.

Common Mistakes with namenode safemode

1. Forgetting that lazy evaluation defers computation until the value is forced, causing space leaks with unevaluated thunks
2. Using return to exit a function early instead of wrapping a pure value in the monad
3. Mixing let bindings with <- bindings in do notation, producing type errors

These mistakes appear frequently in real-world HADOOP code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ