Logging and Auditing for APIs — Complete Security Monitoring Guide
In this tutorial, you will learn about Logging and Auditing for APIs. We cover key concepts, practical examples, and best practices to help you master this topic.
Logging and auditing record all API activity including requests, responses, authentication attempts, and security events. This audit trail is essential for detecting attacks, investigating incidents, and meeting Compliance requirements.
What You'll Learn
You'll learn what to log, how to structure logs for security analysis, and how to implement audit trails that comply with SOC 2, PCI DSS, and HIPAA.
Why It Matters
Without logging, you cannot detect ongoing attacks, investigate breaches, or prove compliance. Logs are the first thing investigators request after a security incident.
Real-World Use
A SOC team detected a brute-force attack on a login API by analyzing rate-of-rise in 401 errors in their ELK Stack logs. They blocked the attacker's IP within 2 minutes of the attack starting.
flowchart LR
A[API Request] --> B[Log Event]
B --> C[Structured JSON]
C --> D[Log Aggregator]
D --> E[Elasticsearch]
E --> F[Grafana/Kibana]
F --> G[Alerts]
E --> H[Security Audit]
H --> I[Compliance Reports]
Teacher's Mindset
API logging is like a security camera system. You install cameras (logging), store footage (log retention), review when incidents occur (forensics), and periodically check for suspicious activity (monitoring).
Implementing Security Logging
import logging
import json
from datetime import datetime
from flask import Flask, request, g
import uuid
app = Flask(__name__)
class SecurityLogger:
def __init__(self):
self.logger = logging.getLogger("api_security")
self.logger.setLevel(logging.INFO)
handler = logging.FileHandler("api-audit.log")
formatter = logging.Formatter(
'{"time": "%(asctime)s", "level": "%(levelname)s", "message": %(message)s}'
)
handler.setFormatter(formatter)
self.logger.addHandler(handler)
def log_request(self, request, user_id=None):
log_entry = {
"request_id": str(uuid.uuid4()),
"timestamp": datetime.utcnow().isoformat(),
"method": request.method,
"path": request.path,
"query": dict(request.args),
"ip": request.remote_addr,
"user_agent": request.headers.get("User-Agent"),
"user_id": user_id,
"content_type": request.content_type,
"content_length": request.content_length
}
self.logger.info(json.dumps(log_entry))
return log_entry["request_id"]
def log_auth_event(self, success, user_id, reason=None):
log_entry = {
"event_type": "authentication",
"timestamp": datetime.utcnow().isoformat(),
"success": success,
"user_id": user_id,
"reason": reason,
"ip": request.remote_addr
}
level = logging.INFO if success else logging.WARNING
self.logger.log(level, json.dumps(log_entry))
security_logger = SecurityLogger()
# Structured JSON logging middleware
import structlog
structlog.configure(
processors=[
structlog.stdlib.add_log_level,
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.JSONRenderer()
],
context_class=structlog.thread_safe.wrap_dict(dict),
logger_factory=structlog.PrintLoggerFactory(),
)
log = structlog.get_logger()
@app.before_request
def log_request_start():
log.info("request_start",
method=request.method,
path=request.path,
ip=request.remote_addr
)
@app.after_request
def log_request_end(response):
log.info("request_end",
status_code=response.status_code,
content_length=response.content_length
)
return response
# Audit trail for sensitive operations
AUDIT_LOG = []
def audit(event_type, user_id, resource, action, success, details=None):
entry = {
"timestamp": datetime.utcnow().isoformat(),
"event_type": event_type,
"user_id": user_id,
"resource": resource,
"action": action,
"success": success,
"details": details,
"ip": request.remote_addr,
"request_id": g.request_id
}
AUDIT_LOG.append(entry)
log.info("audit_event", **entry)
@app.route("/api/users/<user_id>/role", methods=["PUT"])
def update_role(user_id):
audit("role_change", current_user.id, user_id, "update_role", True)
return jsonify({"message": "Role updated"})
Common Mistakes
| Mistake | Why It's Wrong | Fix |
|---|---|---|
| Logging sensitive data (passwords, tokens) | Credential exposure in logs | Implement log scrubbing for sensitive fields |
| No log correlation ID | Cannot trace a request across services | Add unique request ID per request |
| Insufficient log retention | Logs deleted before incident detected | Follow compliance: PCI DSS (1yr), SOC 2 (1yr), HIPAA (6yr) |
| Not logging auth failures | Cannot detect brute-force attacks | Always log failed and successful auth attempts |
| No log integrity protection | Attackers can modify logs to hide tracks | Use append-only storage and log signing |
Practice Questions
- What should be included in every API log entry?
- Why should logs be structured (JSON) rather than plain text?
- What is log correlation and why is it important?
- How do you prevent sensitive data from appearing in logs?
- What is the difference between logging and monitoring?
Challenge
Implement structured JSON logging for a Flask API. Include request ID correlation, auth event logging, and sensitive data scrubbing. Set up log rotation and retention policies.
FAQ
Mini Project
Build a Flask API with comprehensive logging: structured JSON, request ID per request, auth event logging, sensitive data scrubbing, and log rotation. Set up the ELK stack (Elasticsearch, Logstash, Kibana) to ingest and visualize logs.
What's Next
Learn about security Incident Response for effectively handling API security breaches.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro