Lambda + S3 — Serverless File Processing
In this tutorial, you will learn about Lambda + S3. We cover key concepts, practical examples, and best practices to help you master this topic.
AWS S3 event notifications invoke Lambda functions when objects are created, deleted, or restored, enabling Serverless file processing pipelines that scale automatically.
What You'll Learn
By the end of this lesson you will understand how to configure S3 event notifications, Process uploaded files with Lambda, generate thumbnails, validate file content, and scan for malware.
Why It Matters
File processing is one of the most common serverless use cases. S3-to-Lambda pipelines handle image resizing, video transcoding, document conversion, and security scanning -- all without managing any infrastructure.
Real-World Use
Doda Browser's file upload feature uses S3 events to trigger a Lambda function that validates file type and size, scans for malware signatures, generates thumbnail previews, and stores metadata in DynamoDB.
flowchart TD
U[User Upload] --> S3[S3 Bucket]
S3 -->|Event Notification| L[AWS Lambda]
L --> V[Validate File Type]
V --> M[Malware Scan]
M --> T[Generate Thumbnail]
T --> D[Store Metadata in DynamoDB]
T --> S3T[Save Thumbnail to S3]
style L fill:#f90,color:#fff
S3 Event Configuration
S3 can send events to Lambda on object creation (PutObject, PostObject, CopyObject, CompleteMultipartUpload), object deletion, and object restoration.
# s3_event_config.py
# Processing S3 event notifications
import json
def lambda_handler(event, context):
for record in event["Records"]:
bucket = record["s3"]["bucket"]["name"]
key = record["s3"]["object"]["key"]
size = record["s3"]["object"]["size"]
event_name = record["eventName"]
print(f"[S3] Event: {event_name}")
print(f"[S3] Bucket: {bucket}")
print(f"[S3] Key: {key}")
print(f"[S3] Size: {size} bytes")
if key.endswith((".jpg", ".png", ".gif")):
process_image(bucket, key)
elif key.endswith(".mp4"):
process_video(bucket, key)
else:
print(f" -> Unknown file type, skipping processing")
def process_image(bucket, key):
print(f" -> Resizing image: {key}")
print(f" -> Saving thumbnail to thumbnails/{key}")
def process_video(bucket, key):
print(f" -> Transcoding video: {key}")
print(f" -> Saving to transcoded/{key}")
s3_event = {"Records": [{"eventName": "ObjectCreated:Put", "s3": {"bucket": {"name": "uploads"}, "object": {"key": "photos/sunset.jpg", "size": 2048000}}}]}
lambda_handler(s3_event, None)
Expected output:
[S3] Event: ObjectCreated:Put
[S3] Bucket: uploads
[S3] Key: photos/sunset.jpg
[S3] Size: 2048000 bytes
-> Resizing image: photos/sunset.jpg
-> Saving thumbnail to thumbnails/photos/sunset.jpg
Image Processing Pipeline
A common pattern is downloading an image from S3, processing it in memory, and uploading the result back to another S3 location.
# image_processing.py
# Image processing Lambda
import json
import io
def lambda_handler(event, context):
for record in event["Records"]:
bucket = record["s3"]["bucket"]["name"]
key = record["s3"]["object"]["key"]
print(f"Processing s3://{bucket}/{key}")
# Simulate download
image_data = simulate_download(bucket, key)
print(f" Downloaded {len(image_data)} bytes")
# Simulate processing
thumbnail = simulate_thumbnail(image_data)
print(f" Generated thumbnail: {len(thumbnail)} bytes")
# Simulate upload
thumbnail_key = f"thumbnails/{key.split('/')[-1]}"
simulate_upload(bucket, thumbnail_key, thumbnail)
print(f" Uploaded to s3://{bucket}/{thumbnail_key}")
# Return metadata
metadata = {
"originalKey": key,
"thumbnailKey": thumbnail_key,
"size": len(image_data),
"thumbnailSize": len(thumbnail)
}
print(f" Metadata: {json.dumps(metadata)}")
def simulate_download(bucket, key):
return b"mock_image_data" * 1000
def simulate_thumbnail(data):
return data[:1000]
def simulate_upload(bucket, key, data):
pass
lambda_handler({"Records": [{"s3": {"bucket": {"name": "images"}, "object": {"key": "2026/06/sunset.jpg"}}}]}, None)
Expected output:
Processing s3://images/2026/06/sunset.jpg
Downloaded 15000 bytes
Generated thumbnail: 1000 bytes
Uploaded to s3://images/thumbnails/sunset.jpg
Metadata: {"originalKey": "2026/06/sunset.jpg", "thumbnailKey": "thumbnails/sunset.jpg", "size": 15000, "thumbnailSize": 1000}
File Validation and Security Scanning
Before processing user-uploaded files, validate their type, scan for malware, and check for sensitive content.
# file_validation.py
# File validation and security scanning
import json
MALWARE_SIGNATURES = [b"malware_code", b"virus_signature", b"exploit_payload"]
BLOCKED_EXTENSIONS = {".exe", ".bat", ".cmd", ".scr", ".vbs", ".dll"}
def validate_file(key, data):
ext = "." + key.rsplit(".", 1)[1].lower() if "." in key else ""
if ext in BLOCKED_EXTENSIONS:
return {"valid": False, "reason": f"Blocked file type: {ext}"}
for sig in MALWARE_SIGNATURES:
if sig in data:
return {"valid": False, "reason": "Malware signature detected"}
if len(data) > 10 * 1024 * 1024:
return {"valid": False, "reason": "File exceeds 10MB limit"}
return {"valid": True, "reason": "File passed validation"}
def lambda_handler(event, context):
for record in event["Records"]:
key = record["s3"]["object"]["key"]
data = b"safe_content_here"
result = validate_file(key, data)
if result["valid"]:
print(f"[ACCEPT] {key}: {result['reason']}")
else:
print(f"[REJECT] {key}: {result['reason']}")
print(f" -> Moving to quarantine/quarantined/{key}")
test_files = ["photo.jpg", "script.exe", "document.pdf"]
for f in test_files:
event = {"Records": [{"s3": {"object": {"key": f}}}]}
lambda_handler(event, None)
Expected output:
[ACCEPT] photo.jpg: File passed validation
[REJECT] script.exe: Blocked file type: .exe
-> Moving to quarantine/quarantined/script.exe
[ACCEPT] document.pdf: File passed validation
Signed URLs for Secure Upload
Use pre-signed URLs to allow clients to upload directly to S3 without exposing AWS credentials.
# signed_urls.py
# Generating pre-signed URLs
import json
def generate_presigned_upload_url(bucket, key, expires_in=3600):
"""Simulate generating a pre-signed URL."""
url = f"https://{bucket}.s3.amazonaws.com/{key}?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires={expires_in}"
print(f"Generated pre-signed URL for s3://{bucket}/{key}")
print(f" URL expires in {expires_in} seconds")
return url
def lambda_handler(event, context):
body = json.loads(event.get("body", "{}"))
filename = body.get("filename", f"uploads/{event['requestContext']['authorizer']['claims']['sub']}/file")
upload_url = generate_presigned_upload_url("my-app-uploads", filename)
return {
"statusCode": 200,
"body": json.dumps({"uploadUrl": upload_url, "publicUrl": f"https://my-app-uploads.s3.amazonaws.com/{filename}"})
}
mock_event = {"body": json.dumps({"filename": "images/profile.jpg"}), "requestContext": {"authorizer": {"claims": {"sub": "user123"}}}}
print(lambda_handler(mock_event, None)["body"])
Common Mistakes
Not filtering S3 event prefixes: A bucket with multiple use cases invokes Lambda for every object. Use prefix and suffix filters in the event notification.
Recursive S3 triggers: Lambda processing a file and writing output to the same bucket can trigger itself again. Use separate input/output buckets or prefixes.
Forgetting S3 consistency: S3 is eventually consistent for some operations. New objects are immediately consistent, but overwrites take time.
Ignoring large file limits: Lambda has a 15-minute timeout and 10GB memory. For large files, use S3 multipart upload with Step Functions.
Not handling S3 event retries: S3 events may be delivered multiple times. Use idempotent processing with object metadata checks.
Practice Questions
How do you trigger Lambda from S3 events? Configure S3 event notifications on the bucket for specific event types (s3:ObjectCreated:, s3:ObjectRemoved:).
Why should you use separate buckets for input and output? To prevent recursive Lambda invocations when the function writes processed files back to S3.
How do you validate uploaded file types? Check the file extension and file signature (magic bytes) to prevent spoofing, not just the Content-Type header.
What is a pre-signed URL? A temporary URL that grants time-limited access to an S3 object, used for secure uploads without exposing credentials.
Challenge: Design a serverless video processing pipeline with S3, Lambda, and Step Functions that handles upload, transcoding, thumbnail generation, and notification.
FAQ
Mini Project
Create a Lambda function that receives S3 events for uploaded images, validates they are JPEG or PNG (by checking magic bytes), generates a thumbnail, stores it in a processed/ prefix, and returns metadata.
import json
VALID_HEADERS = {
b"\xff\xd8\xff": "JPEG",
b"\x89PNG\r\n\x1a\n": "PNG"
}
def get_file_format(data):
for header, fmt in VALID_HEADERS.items():
if data[:len(header)] == header:
return fmt
return None
def lambda_handler(event, context):
for record in event["Records"]:
key = record["s3"]["object"]["key"]
data = b"\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00"
fmt = get_file_format(data)
if not fmt:
print(f"[REJECT] {key}: Unknown format")
continue
print(f"[ACCEPT] {key}: Detected {fmt}")
print(f" -> Generating thumbnail for thumbnails/{key}")
print(f" -> Saving metadata to DynamoDB")
lambda_handler({"Records": [{"s3": {"object": {"key": "photo.jpg"}}}]}, None)
What's Next
Next: Lambda + SQS for message-driven patterns.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro