How to Fix GitLab Praefect Replication Error

In this tutorial, you'll learn about How to Fix GitLab Praefect Replication Error. We cover key concepts, practical examples, and best practices.

GitLab Praefect replication shows Unhealthy nodes detected or Replication job failed on the /praefect/health endpoint — Gitaly nodes are not synchronizing correctly through the Praefect proxy layer.

The Problem

praefect_1  | time="..." level=error msg="replication job failed"
praefect_1  | virtual_storage: "default", relative_path: "@hashed/xx/yy.git"
praefect_1  | error: "rpc error: code = Internal desc = target node unhealthy"

Step-by-Step Fix

Step 1: Check node health

sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml \
  dial-nodes

Step 2: List virtual storage status

sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml \
  list-untracked-repositories

Step 3: Force a replication

sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml \
  replicate-repository -virtual-storage default \
  -relative-path @hashed/xx/yy.git -target gitaly-2

Step 4: Check Praefect logs

sudo journalctl -u gitlab-praefect -n 100

Step 5: Set a maintenance window and reconcile

sudo gitlab-rake gitlab:praefect:reconcile

Prevention Tips

• Run at least 3 Gitaly nodes per virtual storage for quorum
• Monitor Praefect health with Prometheus metrics
• Set praefect['failover_enabled'] = true in gitlab.rb
• Configure replication worker count based on repository count

Common Mistakes with praefect error

1. Overlapping type class instances that cause GHC to reject the program with ambiguous dispatch errors
2. Non-exhaustive pattern matches that compile with warnings then crash at runtime
3. Misunderstanding that String is [Char] with poor performance for large text operations

These mistakes appear frequently in real-world GITLAB code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ