Python CSV, JSON, and YAML — Complete Guide
In this tutorial, you will learn about Python CSV, JSON, and YAML. We cover key concepts, practical examples, and best practices to help you master this topic.
Real-world Python programs rarely work with just text files. They process CSV exports from databases, JSON responses from APIs, and YAML configuration files. Each format has its own quirks, and Python's standard library handles the most common ones elegantly.
In this tutorial, you'll learn to read, write, and manipulate CSV, JSON, and YAML files — skills you'll use daily in data processing, API development, and configuration management. DodaTech tools process threat intelligence feeds in CSV, expose REST APIs in JSON, and read tool configuration in YAML.
What You'll Learn
- CSV: reading, writing, DictReader, handling edge cases
- JSON: Parsing, generating, pretty-printing, custom Serialization
- YAML: loading, dumping, safety considerations
- Converting between formats
- Handling large files without loading everything into memory
Why File Formats Matter
Data doesn't come in one format. A typical pipeline at DodaTech might: read a CSV of IP addresses from a threat feed, enrich each entry by querying a JSON API, and save the results as YAML configuration for firewall rules. You need to be fluent in all three.
CSV — Comma-Separated Values
Python's csv module handles the complexities of CSV (quoted fields, escaped commas, different dialects):
import csv
# Reading CSV into lists
with open("employees.csv", "w", newline="") as f:
writer = csv.writer(f)
writer.writerow(["Name", "Department", "Salary"])
writer.writerow(["Alice", "Engineering", 85000])
writer.writerow(["Bob", "Marketing", 65000])
writer.writerow(["Charlie", "Engineering", 95000])
# Read it back
with open("employees.csv", newline="") as f:
reader = csv.reader(f)
for row in reader:
print(row)
Expected output:
['Name', 'Department', 'Salary']
['Alice', 'Engineering', '85000']
['Bob', 'Marketing', '65000']
['Charlie', 'Engineering', '95000']
DictReader and DictWriter
Much more readable — access columns by name:
with open("employees.csv", newline="") as f:
reader = csv.DictReader(f)
for row in reader:
print(f"{row['Name']} in {row['Department']} earns ${row['Salary']}")
# Calculate average salary by department
with open("employees.csv", newline="") as f:
reader = csv.DictReader(f)
dept_totals = {}
dept_counts = {}
for row in reader:
dept = row["Department"]
salary = int(row["Salary"])
dept_totals[dept] = dept_totals.get(dept, 0) + salary
dept_counts[dept] = dept_counts.get(dept, 0) + 1
for dept, total in dept_totals.items():
avg = total / dept_counts[dept]
print(f"{dept} average salary: ${avg:.0f}")
Expected output:
Alice in Engineering earns $85000
Bob in Marketing earns $65000
Charlie in Engineering earns $95000
Engineering average salary: $90000
Marketing average salary: $65000
Handling Edge Cases
import csv
# Fields with commas must be quoted
data = [
["Alice", "Engineering", "$85,000"], # Comma in value
["Bob", "Marketing", "$65,000"],
]
with open("salaries.csv", "w", newline="") as f:
writer = csv.writer(f)
writer.writerow(["Name", "Dept", "Salary"])
writer.writerows(data)
# Read it back — the csv module handles quoting automatically
with open("salaries.csv", newline="") as f:
for row in csv.reader(f):
print(row)
# Different delimiters (TSV — tab-separated)
with open("data.tsv", "w", newline="") as f:
writer = csv.writer(f, delimiter="\t")
writer.writerow(["Name", "Score"])
writer.writerow(["Alice", 95])
with open("data.tsv", newline="") as f:
for row in csv.reader(f, delimiter="\t"):
print(row)
JSON — JavaScript Object Notation
JSON is the lingua franca of web APIs. Python's json module makes it trivial:
import json
# Python object -> JSON string
data = {
"name": "Alice",
"age": 30,
"skills": ["Python", "Docker", "Kubernetes"],
"active": True,
"salary": None
}
json_string = json.dumps(data, indent=2)
print(json_string)
Expected output:
{
"name": "Alice",
"age": 30,
"skills": ["Python", "Docker", "Kubernetes"],
"active": true,
"salary": null
}
Reading JSON from APIs
import json
from urllib.request import urlopen
# Simulate an API response
response = json.dumps({
"status": "ok",
"data": {
"users": [
{"id": 1, "name": "Alice"},
{"id": 2, "name": "Bob"}
]
}
})
parsed = json.loads(response)
print(parsed["data"]["users"][0]["name"]) # Alice
# Writing JSON to a file
with open("output.json", "w") as f:
json.dump(data, f, indent=2)
# Reading JSON from a file
with open("output.json") as f:
loaded = json.load(f)
print(loaded["name"]) # Alice
Custom JSON Serialization
Not everything is JSON-serializable by default:
import json
from datetime import datetime
class User:
def __init__(self, name, joined):
self.name = name
self.joined = joined
# This fails: TypeError: Object of type User is not JSON serializable
def custom_serializer(obj):
if isinstance(obj, datetime):
return obj.isoformat()
if isinstance(obj, User):
return {"name": obj.name, "joined": obj.joined.isoformat()}
raise TypeError(f"Can't serialize {type(obj)}")
user = User("Alice", datetime(2026, 6, 29))
print(json.dumps(user, default=custom_serializer, indent=2))
Expected output:
{
"name": "Alice",
"joined": "2026-06-29T00:00:00"
}
YAML — Yet Another Markup Language
YAML is popular for configuration files. Python needs a third-party library (pyyaml):
import yaml
# Reading YAML
config_yaml = """
server:
host: localhost
port: 8080
database:
url: postgresql://localhost/mydb
pool_size: 10
features:
- logging
- monitoring
- caching
"""
config = yaml.safe_load(config_yaml)
print(config["server"]["host"]) # localhost
print(config["features"]) # ['logging', 'monitoring', 'caching']
print(config["database"]["pool_size"]) # 10
# Writing YAML
output = yaml.dump(config, default_flow_style=False)
print(output)
Security: Never Use yaml.load()
import yaml
# DANGEROUS: yaml.load() can execute arbitrary code
# Never use yaml.load() on untrusted input!
dangerous = "!!python/object/apply:os.system ['rm -rf /']"
# yaml.load(dangerous) # Would execute the command!
# SAFE: yaml.safe_load() only parses standard YAML
safe = yaml.safe_load(dangerous) # raises yaml.constructor.ConstructorError
Converting Between Formats
Real pipelines often convert between formats:
import csv, json
# CSV -> JSON
def csv_to_json(csv_path, json_path):
data = []
with open(csv_path, newline="") as f:
reader = csv.DictReader(f)
for row in reader:
data.append(row)
with open(json_path, "w") as f:
json.dump(data, f, indent=2)
csv_to_json("employees.csv", "employees.json")
Common Pitfalls
Pitfall 1: CSV with BOM
Some Excel CSV files start with a BOM (Byte Order Mark):
# Handle BOM by using encoding="utf-8-sig"
with open("excel_export.csv", newline="", encoding="utf-8-sig") as f:
reader = csv.reader(f)
for row in reader:
print(row) # First field won't have \ufeff prefix
Pitfall 2: JSON with Trailing Commas
JSON doesn't allow trailing commas, but humans add them:
# Invalid JSON
bad_json = '{"name": "Alice", "age": 30,}'
# json.loads(bad_json) # json.decoder.JSONDecodeError
# Fix: strip trailing comma before closing brace
import re
fixed = re.sub(r",\s*}", "}", bad_json)
print(json.loads(fixed)) # Works!
Pitfall 3: Large JSON Files
Don't load huge JSON files into memory — use streaming:
# BAD: loads entire file
with open("huge.json") as f:
data = json.load(f) # MemoryError for large files
# GOOD: stream using ijson (third party)
# Or restructure the file as JSON Lines (.jsonl) — one JSON object per line
def read_jsonl(path):
with open(path) as f:
for line in f:
if line.strip():
yield json.loads(line)
for record in read_jsonl("huge_data.jsonl"):
process(record) # One record at a time
Practice Questions
Write a function that reads a CSV and returns only rows where a specific column matches a value.
Convert a nested JSON structure to a flat CSV (one row per leaf value).
Write a YAML config validator that checks required fields exist and have correct types.
Merge two JSON files — if a key exists in both, the second file's value wins.
Write a CSV dialect detector: given a CSV file, detect the delimiter, quote character, and whether it has a header row.
Challenge: ETL Pipeline
Build a pipeline that:
- Reads a CSV of product data (id, name, category, price, stock)
- Enriches each product by looking up the category name from a JSON file
- Filters out products with zero stock
- Groups by category and calculates average price
- Outputs the result as both CSV and YAML
Real-World Task: Log Converter
DodaTech's security tools generate logs in multiple formats. Write a function that converts Apache common log format (space-separated) to JSON. Each log line looks like:
192.168.1.1 - frank [10/Oct/2026:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326
Parse it into: ip, ident, user, timestamp, method, path, protocol, status, bytes. Output as JSON.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro