Lambda Layers — Managing Shared Dependencies

In this tutorial, you will learn about Lambda Layers. We cover key concepts, practical examples, and best practices to help you master this topic.

AWS Lambda Layers let you package libraries, custom runtimes, and shared code separately from your function code, reducing deployment size and enabling dependency reuse across functions.

What You'll Learn

By the end of this lesson you will understand how to create Lambda Layers, attach them to functions, manage layer versions, and organize shared code for multi-function projects.

Why It Matters

Without layers, every Lambda function must include all its dependencies in the deployment package. A 50MB package with Pillow or Pandas increases cold start time and makes updates tedious. Layers separate dependencies from code so you update each independently.

Real-World Use

DodaTech's file processing pipeline has multiple Lambda functions for scanning, resizing, and watermarking images. A shared layer contains the Pillow library and custom image utilities -- a security update to the library updates all functions simultaneously.

flowchart LR
    L[Shared Layer] --> F1[Function: Scan]
    L --> F2[Function: Resize]
    L --> F3[Function: Watermark]
    L --> F4[Function: Compress]
    L4[Layer: shared-utils v5]
    F1 --> D[Deploy: just function code]
    F2 --> D
    F3 --> D
    F4 --> D
    style L fill:#f90,color:#fff

Creating a Layer

A layer is a ZIP archive containing dependencies in specific folders. Python libraries go in python/lib/python3.x/site-packages/. Node.js modules go in nodejs/node_modules/.

# Create a Python layer with requests library
mkdir -p layer/python/lib/python3.9/site-packages/
pip install requests -t layer/python/lib/python3.9/site-packages/
cd layer && zip -r ../requests-layer.zip .

# function.py
# Using the requests library from a layer

import json
import requests

def lambda_handler(event, context):
    url = event.get("url", "https://api.github.com")
    response = requests.get(url, timeout=5)
    
    return {
        "statusCode": response.status_code,
        "body": json.dumps({
            "url": url,
            "status": response.status_code,
            "headers": dict(response.headers)
        })
    }

test_event = {"url": "https://api.github.com"}
print(json.dumps(lambda_handler(test_event, None), indent=2))

Expected output:

{
  "statusCode": 200,
  "body": "{\"url\": \"https://api.github.com\", \"status\": 200, \"headers\": {...}}"
}

Layer Version Management

Each layer publish creates an immutable version. Functions reference a specific layer version by ARN. You can also use $LATEST for development but pin versions in production.

# layer_versions.py
# Managing layer versions

def simulate_layer_deployment(layer_name, version, libraries):
    print(f"Publishing {layer_name} v{version}")
    for lib in libraries:
        print(f"  Including: {lib}")
    layer_arn = f"arn:aws:lambda:us-east-1:123456789012:layer:{layer_name}:{version}"
    print(f"  ARN: {layer_arn}")
    return layer_arn

def update_function_layer(function_name, layer_arn):
    print(f"Updating {function_name} to use {layer_arn}")
    print(f"  -> Function updated successfully")

# Initial deployment
v1 = simulate_layer_deployment("shared-utils", 1, ["requests==2.31.0", "pyyaml==6.0"])
update_function_layer("process-images", v1)

# Security update
v2 = simulate_layer_deployment("shared-utils", 2, ["requests==2.32.0", "pyyaml==6.1"])
update_function_layer("process-images", v2)

Expected output:

Publishing shared-utils v1
  Including: requests==2.31.0
  Including: pyyaml==6.0
  ARN: arn:aws:lambda:us-east-1:123456789012:layer:shared-utils:1
Updating process-images to use arn:aws:lambda:...:shared-utils:1
  -> Function updated successfully
Publishing shared-utils v2
  Including: requests==2.32.0
  Including: pyyaml==6.1
  ARN: arn:aws:lambda:us-east-1:123456789012:layer:shared-utils:2
Updating process-images to use arn:aws:lambda:...:shared-utils:2
  -> Function updated successfully

Multiple Layers

Functions can use up to five layers. Layer contents are merged into the /opt directory in the execution environment. Order matters -- layers are applied in sequence.

# multiple_layers.py
# Working with multiple layers

def describe_layer_contents():
    layers = {
        "base-layer v1": ["python/", "lib/"],
        "utils-layer v3": ["python/lib/python3.9/site-packages/requests/"],
        "custom-layer v2": ["python/lib/python3.9/site-packages/myapp/"],
    }
    
    total_size = 0
    for layer_name, contents in layers.items():
        size = len(layer_name) * 100000
        total_size += size
        print(f"{layer_name}: {contents}")
    
    print(f"\nTotal mounted at /opt/: ~{total_size}KB")
    print("All layers merged into single filesystem at /opt/")

describe_layer_contents()

Custom Runtimes with Layers

Layers can contain custom runtimes for languages not natively supported by Lambda. The runtime must include a Bootstrap file that handles the Lambda runtime API.

# custom_runtime.py
# Custom runtime bootstrap concept

def custom_runtime_bootstrap():
    print("[Runtime] Starting custom runtime...")
    print("[Runtime] Waiting for invocation...")
    print("[Runtime] Received /runtime/invocation/next")
    print("[Runtime] Processing event...")
    print("[Runtime] Posting response to /runtime/invocation/response")
    print("[Runtime] Done")

custom_runtime_bootstrap()

Common Mistakes

1. Including unnecessary dependencies: Each layer adds to the total deployment size. Only include what the function actually needs.

2. Using absolute paths in layer code: Layer code is extracted to /opt. Use relative imports or sys.path modifications for custom modules.

3. Mixing incompatible runtime versions: A Python 3.9 layer does not work with a Python 3.11 function. Match runtime versions precisely.

4. Forgetting layer limits: Maximum of five layers per function. Total unzipped size including all layers is 250MB.

5. Not versioning layers properly: Always pin specific layer versions in production. Using $LATEST can break deployments when layers are updated.

Practice Questions

1. What is the maximum number of layers per Lambda function? Five layers per function. Total unzipped deployment package including layers cannot exceed 250MB.

2. Where are layer contents extracted in the execution environment? The /opt directory. Python layers go in /opt/python/, Node.js in /opt/nodejs/.

3. Why use layers instead of including dependencies in the function package? Layers enable sharing across functions, independent versioning, smaller function packages, and faster deployments.

4. Can a layer be shared across multiple AWS accounts? Yes. Layers can be shared across accounts using resource-based policies or through AWS Organizations.

5. Challenge: Design a layer Strategy for a project with 10 Lambda functions that share 60% common code but need 40% function-specific dependencies.

FAQ

Mini Project

Create three layers for a Lambda project: a base layer with common utilities, an AWS SDK layer, and a business logic layer. Write a function that uses all three.

import json

# Simulating layer imports
print("[Base Layer] Initializing logging and config...")
print("[AWS Layer] Initializing boto3 clients...")
print("[Business Layer] Loading business rules...")

def lambda_handler(event, context):
    data = event.get("data", {})
    result = process_business_rule(data)
    return {"statusCode": 200, "body": json.dumps(result)}

def process_business_rule(data):
    print(f"Processing: {data}")
    return {"processed": True, "output": data}

result = lambda_handler({"data": {"type": "order", "amount": 100}}, None)
print(json.dumps(result))

What's Next

Next: Lambda Environment Variables for Configuration Management.