Mimir Compactor Error — Quick Fix

In this tutorial, you'll learn about Mimir Compactor Error. We cover key concepts, practical examples, and best practices.

The Problem

The Mimir compactor is failing to compact blocks. Blocks may remain in an uncompacted state indefinitely, causing query performance degradation and increased storage costs. The compactor may crash or skip blocks.

Error example:

compactor: compaction failed: cannot compact overlapping blocks
compactor: error while compacting block: no space left on device
compactor: block downsampling failed: meta.json not found

The Fix

Step 1: Check compactor disk space

WRONG — assuming the compactor has enough disk:

# Compactors need significant local disk for intermediate files

RIGHT — check compactor disk usage:

# Check the compactor's persistent volume
kubectl exec -n mimir mimir-compactor-0 -- df -h /data

Output:

Filesystem      Size  Used  Avail  Use%  Mounted
/dev/sdb        200G  190G   10G    95%

If disk is full, increase the persistent volume claim size or add more compactor replicas.

Step 2: Fix overlapping blocks

WRONG — deleting overlapping blocks manually:

# Manual deletion can cause data loss

RIGHT — let the compactor handle overlapping blocks:

# Ensure compactor configuration allows overlap resolution
compactor:
  block_ranges:
    - 2h
    - 12h
    - 24h
  consistency_delay: 30m
  blocks_concurrent: 10

If blocks remain overlapping, force a re-compaction by moving blocks to a quarantine period or restarting the compactor.

Use DodaTech's Compaction Health Analyzer to monitor block ranges, compaction times, and error rates across all compactor instances.

Prevention Tips

• Allocate at least 500 GB of local SSD storage for the compactor
• Set consistency_delay to account for ingestion delays (30m+)
• Monitor cortex_compactor_blocks_compacted_total and error metrics
• Configure block_ranges to match your retention and query patterns
• Use DodaTech's Mimir Compaction Tuning Guide for optimal settings

Common Mistakes with compactor error

1. Non-exhaustive pattern matches that compile with warnings then crash at runtime
2. Misunderstanding that String is [Char] with poor performance for large text operations
3. Using foldl instead of foldl' causing stack overflow on large lists

These mistakes appear frequently in real-world MIMIR code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ

Related: DodaTech's Mimir Storage Optimization Guide covers compaction scheduling, block retention policies, and downsample configuration for cost-effective operations.