MongoDB Aggregation Pipeline Deep Dive

DodaTech Updated 2026-06-22 9 min read

MongoDB aggregation pipeline is a framework for data processing that passes documents through a sequence of stages -- each stage transforms the data before passing it to the next -- enabling filtering, grouping, joining, computing, and reshaping documents within the database.

What You'll Learn

You will understand how to build efficient aggregation pipelines using $match, $group, $project, $sort, $lookup, $unwind, $bucket, $facet, and $addFields. You will learn stage ordering for performance, when to use allowDiskUse, and how to debug slow pipelines.

Why Aggregation Matters

Application-level data processing is slow and wasteful. DodaZIP analyzes millions of compressed archives. Using the aggregation pipeline to compute file type statistics, archive sizes, and compression ratios in the database reduced processing time from minutes to seconds.

Aggregation Learning Path

flowchart LR
  A[MongoDB Basics] --> B[CRUD Operations]
  B --> C[Aggregation Pipeline]
  C --> D[NoSQL Data Modeling]
  C:::current
  classDef current fill:#f90,color:#fff,stroke:#333,stroke-width:2px

ℹ️ Info

Prerequisites: Understanding of basic MongoDB CRUD operations. Familiarity with JSON document structure and JavaScript syntax.

Pipeline Stage Overview

Each stage receives documents, transforms them, and outputs to the next stage.

Stage	Purpose	Behavior
`$match`	Filter documents	Reduces document count early
`$project`	Reshape documents	Select/include/exclude fields
`$group`	Aggregate values	Group by key, compute accumulators
`$sort`	Order documents	Memory or disk-based
`$lookup`	Join collections	Left outer join
`$unwind`	Deconstruct arrays	Creates one doc per array element
`$bucket`	Group into ranges	Histogram creation
`$facet`	Multi-pipeline	Multiple aggregations in one pass
`$addFields`	Compute new fields	Add computed fields to documents
`$set`	Alias for $addFields	Same as $addFields

Basic Aggregation Pipeline

Simple Filter and Group

db.orders.aggregate([
    // Stage 1: Filter only shipped orders from 2026
    {
        $match: {
            status: "shipped",
            orderDate: {
                $gte: ISODate("2026-01-01"),
                $lt: ISODate("2027-01-01")
            }
        }
    },
    // Stage 2: Group by customer and compute totals
    {
        $group: {
            _id: "$customerId",
            totalSpent: { $sum: "$total" },
            orderCount: { $sum: 1 },
            avgOrderValue: { $avg: "$total" }
        }
    },
    // Stage 3: Sort by total spent descending
    {
        $sort: { totalSpent: -1 }
    },
    // Stage 4: Limit to top 10 customers
    {
        $limit: 10
    }
]);

Expected output:

[
  { "_id": ObjectId("..."), "totalSpent": 54321, "orderCount": 45, "avgOrderValue": 1207.13 },
  { "_id": ObjectId("..."), "totalSpent": 43210, "orderCount": 38, "avgOrderValue": 1137.11 }
]

$lookup: Joining Collections

MongoDB is not a relational database, but $lookup enables left outer joins.

db.orders.aggregate([
    {
        $match: { status: "shipped" }
    },
    {
        $lookup: {
            from: "customers",
            localField: "customerId",
            foreignField: "_id",
            as: "customer]
        }
    },
    // Unwind the customer array (single match expected)
    {
        $unwind: "$customer"
    },
    {
        $project: {
            orderId: "$_id",
            customerName: "$customer.name",
            total: 1,
            orderDate: 1
        }
    }
]);

$lookup with Pipeline

// Complex lookup with sub-pipeline
db.orders.aggregate([
    {
        $lookup: {
            from: "products",
            let: { productIds: "$items.productId" },
            pipeline: [
                {
                    $match: {
                        $expr: { $in: ["$_id", "$$productIds"] }
                    }
                },
                {
                    $project: { name: 1, price: 1, category: 1 }
                }
            ],
            as: "products"
        }
    }
]);

$unwind and $group Patterns

Unwind and Group

// For each order, explode the items array and compute per-product stats
db.orders.aggregate([
    { $match: { status: "shipped" } },
    { $unwind: "$items" },
    {
        $group: {
            _id: "$items.productId",
            totalSold: { $sum: "$items.quantity" },
            revenue: { $sum: { $multiply: ["$items.quantity", "$items.price"] } },
            uniqueCustomers: { $addToSet: "$customerId" }
        }
    },
    {
        $project: {
            totalSold: 1,
            revenue: 1,
            uniqueCustomerCount: { $size: "$uniqueCustomers" }
        }
    },
    { $sort: { revenue: -1 } },
    { $limit: 20 }
]);

$bucket: Histogram Creation

// Create price histogram
db.products.aggregate([
    {
        $bucket: {
            groupBy: "$price",
            boundaries: [0, 10, 25, 50, 100, 250, 500, 1000],
            default: "1000+",
            output: {
                count: { $sum: 1 },
                avgPrice: { $avg: "$price" },
                products: { $push: { name: "$name", price: "$price" } }
            }
        }
    }
]);

Output:

[
  { "_id": 0, "count": 120, "avgPrice": 4.99, "products": [...] },
  { "_id": 10, "count": 340, "avgPrice": 17.50, "products": [...] },
  { "_id": 1000, "count": 5, "avgPrice": 2499.99, "products": [...] }
]

db.orders.aggregate([
    { $match: { status: "shipped" } },
    {
        $facet: {
            // Total revenue and count
            "summary": [
                {
                    $group: {
                        _id: null,
                        totalRevenue: { $sum: "$total" },
                        totalOrders: { $sum: 1 },
                        avgValue: { $avg: "$total" }
                    }
                }
            ],
            // Top 5 customers
            "topCustomers": [
                { $group: { _id: "$customerId", spent: { $sum: "$total" } } },
                { $sort: { spent: -1 } },
                { $limit: 5 }
            ],
            // Orders by month
            "monthlyTrend": [
                {
                    $group: {
                        _id: { $dateToString: { format: "%Y-%m", date: "$orderDate" } },
                        count: { $sum: 1 }
                    }
                },
                { $sort: { _id: 1 } }
            ]
        }
    }
]);

This runs three aggregations in a single pipeline pass, avoiding three separate queries.

$addFields: Computed Fields

db.orders.aggregate([
    {
        $addFields: {
            // Compute order value category
            valueCategory: {
                $switch: {
                    branches: [
                        { case: { $lt: ["$total", 50] }, then: "low" },
                        { case: { $lt: ["$total", 200] }, then: "medium" },
                        { case: { $lt: ["$total", 1000] }, then: "high" }
                    ],
                    default: "premium"
                }
            },
            // Days since order
            daysSinceOrder: {
                $trunc: {
                    $divide: [
                        { $subtract: [new Date(), "$orderDate"] },
                        86400000  // ms in a day
                    ]
                }
            }
        }
    }
]);

Pipeline Performance Optimization

Stage Ordering

Always put $match and $limit as early as possible to reduce the document count before expensive operations like $group and $sort.

// BAD: $group before $match
db.orders.aggregate([
    { $group: { _id: "$customerId", total: { $sum: "$total" } } },
    { $match: { total: { $gt: 1000 } } }  // Groups ALL documents first
]);

// GOOD: $match before $group
db.orders.aggregate([
    { $match: { total: { $gt: 1000 }, status: "shipped" } },
    { $group: { _id: "$customerId", total: { $sum: "$total" } } }
]);

Index Usage

$match and $sort stages can use indexes if they appear early in the pipeline.

// Create index for common aggregation pattern
db.orders.createIndex({ status: 1, orderDate: -1 });

// This pipeline uses the index for $match and $sort
db.orders.aggregate([
    { $match: { status: "shipped", orderDate: { $gte: ISODate("2026-01-01") } } },
    { $sort: { orderDate: -1 } },
    { $group: { _id: "$customerId", orders: { $push: "$$ROOT" } } }
]);

allowDiskUse

Large aggregations may exceed MongoDB's 100MB memory limit for $sort and $group.

db.orders.aggregate(
    [
        { $group: { _id: "$customerId", total: { $sum: "$total" } } },
        { $sort: { total: -1 } }
    ],
    { allowDiskUse: true }  // Enable disk-based processing
);

Common Aggregation Errors

1. Forgetting $match Before $group

Aggregating all documents before filtering processes unnecessary data and may exceed memory limits.

2. Using $lookup Without Index

$lookup performs better when the foreign field is indexed. Always index localField and foreignField.

3. Not Using allowDiskUse for Large Datasets

Pipeline processing has a 100MB memory limit per stage. Enable allowDiskUse for large aggregations to prevent errors.

4. Misunderstanding $unwind on Null/Missing Fields

$unwind removes documents where the array field is null, missing, or empty. Use preserveNullAndEmptyArrays: true to keep them.

$facet runs multiple pipelines in parallel. If one sub-pipeline is expensive (full collection scan), it blocks all others.

6. Not Projecting Before $lookup

$lookup brings the entire foreign document. Use a sub-pipeline with $project to limit fields transferred.

7. Incorrect $group _id Syntax

The _id field in $group defines the grouping key. Use { _id: "$field" } for single key, { _id: { f1: "$f1", f2: "$f2" } } for compound keys.

Practice Questions

1. What is the first stage you should add to optimize any pipeline?

$match to filter documents as early as possible, reducing the number of documents processed by subsequent stages.

2. How does $lookup differ from a SQL JOIN?

$lookup is a left outer join that adds a nested array to each document. It cannot do inner joins or cross joins directly.

3. When should you use allowDiskUse?

When the pipeline processes more than 100MB of data in $sort or $group stages. Without it, MongoDB returns an error.

4. What is the difference between $project and $addFields?

$project controls which fields exist in the output (inclusion/exclusion). $addFields adds new fields while keeping all existing fields.

5. Challenge: Build a sales analytics pipeline.

Given an orders collection with this structure:

{
  "_id": ObjectId,
  "customerId": ObjectId,
  "orderDate": ISODate,
  "status": "shipped",
  "items": [
    { "productId": ObjectId, "quantity": 2, "price": 29.99 }
  ],
  "total": 59.98
}

Build a pipeline that returns monthly revenue by product category. Answer:

db.orders.aggregate([
    { $match: { status: "shipped" } },
    { $unwind: "$items" },
    {
        $lookup: {
            from: "products",
            localField: "items.productId",
            foreignField: "_id",
            as: "product]
        }
    },
    { $unwind: "$product" },
    {
        $group: {
            _id: {
                month: { $dateToString: { format: "%Y-%m", date: "$orderDate" } },
                category: "$product.category"
            },
            revenue: { $sum: { $multiply: ["$items.quantity", "$items.price"] } },
            unitsSold: { $sum: "$items.quantity" }
        }
    },
    { $sort: { "_id.month": 1, revenue: -1 } }
]);

FAQ

Can I use MongoDB aggregation for real-time analytics?

Yes, for real-time dashboards with sub-second latency. For large historical datasets, pre-aggregate with $merge into a summary collection or use a dedicated analytics database.

What is the maximum document size in an aggregation pipeline?

MongoDB documents have a 16MB BSON size limit. If a stage produces documents exceeding 16MB, use $project to reduce field sizes or $unwind to break documents apart.

How do I debug a slow aggregation pipeline?

Use explain("executionStats") on the pipeline. Check totalDocsExamined vs totalDocsReturned. Large ratios indicate missing early filters.

Does $lookup support sharded collections?

Yes, $lookup works across sharded collections in MongoDB 5.0+. Performance depends on shard key alignment between the source and foreign collections.

Try It Yourself

Create an aggregation pipeline for order analysis:

Insert sample data into an orders collection with 10,000 documents
Create a pipeline that filters by status and date
Use $group to compute total revenue per customer
Add $lookup to include customer name
Sort by revenue and limit to top 10
Use explain() to check index usage
Add an index and compare performance

What's Next

MongoDB Explained

NoSQL Data Modeling

Redis

You have mastered the MongoDB aggregation pipeline including $lookup, $facet, $bucket, and optimization techniques. Build a facet pipeline to run multiple analytics queries in one pass and measure the performance improvement.

← Previous MySQL Performance Tuning: Complete Optimization Guide Next → Redis Caching Patterns: Practical Use Cases Guide

Built by the developers of DodaTech

Doda Browser, DodaZIP & Durga Antivirus Pro

Home Browse Databases