Hadoop Kerberos Authentication Error Fix

In this tutorial, you'll learn about Hadoop Kerberos Authentication Error Fix. We cover key concepts, practical examples, and best practices.

Running a Hadoop command fails with:

javax.security.auth.login.LoginException: Clock skew too great

Kerberos requires the client and server clocks to be synchronized within 5 minutes (default). If the time difference exceeds this limit, authentication fails. Other common errors include missing credentials cache, wrong principal name, or unreachable KDC.

Step-by-Step Fix

1. Fix clock skew

WRONG — ignoring time synchronization:

RIGHT — synchronize clocks:

# Install NTP
sudo apt-get install ntp
sudo systemctl start ntp
sudo systemctl enable ntp

# Force immediate sync
sudo ntpdate -u pool.ntp.org

# Check time difference
ntpq -p

Increase the allowed clock skew (temporary):

# In krb5.conf
[libdefaults]
clock_skew = 300  # Default: 300 seconds (5 minutes)

2. Obtain and renew Kerberos ticket

WRONG — expired or missing ticket:

hdfs dfs -ls /  # Fails: Failed to find any Kerberos tgt

RIGHT — get a new ticket:

# Obtain ticket
kinit myuser@EXAMPLE.COM

# Check ticket status
klist

# Renew ticket (if renewable)
kinit -R

# The ticket expires by default in 24 hours
# For long-running jobs, use keytab:
kinit -k -t /path/to/myuser.keytab myuser@EXAMPLE.COM

3. Check principal name

WRONG — wrong principal format:

# Error: Server not found in Kerberos database

RIGHT — verify the principal exists:

# List all principals (requires admin access)
kadmin.local -q "listprincs" | grep myuser

# The format should be:
# myuser@EXAMPLE.COM (for users)
# nn/hostname@EXAMPLE.COM (for services)

Common Hadoop service principals:

HTTP/hostname@EXAMPLE.COM    # Web UI
nn/hostname@EXAMPLE.COM      # NameNode
dn/hostname@EXAMPLE.COM      # DataNode
rm/hostname@EXAMPLE.COM      # ResourceManager

4. Configure Hadoop security properties

<!-- core-site.xml -->
<property>
    <name>hadoop.security.authentication</name>
    <value>kerberos</value>
</property>

<property>
    <name>hadoop.security.authorization</name>
    <value>true</value>
</property>

<!-- hdfs-site.xml -->
<property>
    <name>dfs.namenode.kerberos.principal</name>
    <value>nn/_HOST@EXAMPLE.COM</value>
</property>

<property>
    <name>dfs.datanode.kerberos.principal</name>
    <value>dn/_HOST@EXAMPLE.COM</value>
</property>

5. Configure JAAS login file

# /etc/hadoop/conf/jaas.conf
Client {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="/etc/security/keytabs/hdfs.keytab"
    storeKey=true
    useTicketCache=false
    principal="nn/hostname@EXAMPLE.COM";
};

Reference in hadoop-env.sh:

export HADOOP_OPTS="$HADOOP_OPTS -Djava.security.auth.login.config=/etc/hadoop/conf/jaas.conf"

6. Debug Kerberos issues

# Enable Kerberos debugging
export HADOOP_OPTS="$HADOOP_OPTS -Dsun.security.krb5.debug=true"

# Run the command again — now you'll see detailed Kerberos output:
# >>> KinitOptions cache name is /tmp/krb5cc_1000
# >>> KeyTab: cache is not empty
# >>> Principal is nn/hostname@EXAMPLE.COM

Check the KDC status:

# Check if KDC is running
systemctl status krb5-kdc

# Check KDC logs
tail -f /var/log/krb5kdc.log

Expected output: Hadoop commands authenticate successfully with Kerberos.

Prevention

• Synchronize clocks across all cluster nodes with NTP.
• Use keytabs for services and long-running processes.
• Set up ktutil to rotate keytab passwords periodically.
• Configure ticket_lifetime and renew_lifetime in krb5.conf.
• Monitor Kerberos ticket expiration with alerts.

Common Mistakes with kerberos auth

1. Forgetting that lazy evaluation defers computation until the value is forced, causing space leaks with unevaluated thunks
2. Using return to exit a function early instead of wrapping a pure value in the monad
3. Mixing let bindings with <- bindings in do notation, producing type errors

These mistakes appear frequently in real-world HADOOP code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ

kadmin.local -q "addprinc -randkey nn/hostname@EXAMPLE.COM"
kadmin.local -q "ktadd -k /etc/security/keytabs/nn.keytab nn/hostname@EXAMPLE.COM"
chmod 400 /etc/security/keytabs/nn.keytab
chown hdfs:hadoop /etc/security/keytabs/nn.keytab