Python httpx and aiohttp: Modern HTTP Clients for API Testing
In this tutorial, you will learn about Python httpx and aiohttp: Modern HTTP Clients for API Testing. We cover key concepts, practical examples, and best practices to help you master this topic.
Modern Python HTTP clients like httpx and aiohttp provide async/await support, HTTP/2, connection pooling, streaming, and advanced features beyond the requests library, essential for testing modern async APIs.
What You'll Learn
How to use httpx (sync and async) and aiohttp for API testing, make concurrent requests with async/await, use connection pooling for performance, test Websocket endpoints, handle streaming responses, and choose between httpx and aiohttp.
Why It Matters
Modern APIs increasingly use async patterns, HTTP/2, WebSockets, and streaming. Requests library is synchronous only. httpx and aiohttp support these modern patterns. DodaTech uses httpx for async microservice tests and load testing.
Real-World Use
A DodaTech service calls 3 internal APIs in parallel. Using httpx async client, the test fires 3 concurrent requests, awaits all responses, validates each, and completes in the time of the slowest request (450ms) instead of 3 sequential calls (1.4s).
flowchart LR
A["async\nClient"] --> B["Request 1\nGET /users"]
A --> C["Request 2\nGET /products"]
A --> D["Request 3\nGET /orders"]
B --> E["await\nasyncio.gather()"]
C --> E
D --> E
E --> F["All Responses\nReady"]
F --> G["Assertions"]
style A fill:#dbeafe,stroke:#2563eb
style E fill:#bbf7d0,stroke:#16a34a
httpx Basic Usage
import httpx
import pytest
# Sync client
def test_get_users_sync():
with httpx.Client(base_url="https://api.dodatech.com/v1") as client:
response = client.get("/users")
assert response.status_code == 200
data = response.json()
assert isinstance(data, list)
assert len(data) > 0
print(f"Users: {len(data)}")
# Async client
@pytest.mark.asyncio
async def test_get_users_async():
async with httpx.AsyncClient(base_url="https://api.dodatech.com/v1") as client:
response = await client.get("/users")
assert response.status_code == 200
data = response.json()
assert isinstance(data, list)
Concurrent Requests with httpx
import asyncio
import httpx
async def fetch_user_data(user_id):
async with httpx.AsyncClient() as client:
user_resp = await client.get(
f"https://api.dodatech.com/v1/users/{user_id}"
)
orders_resp = await client.get(
f"https://api.dodatech.com/v1/users/{user_id}/orders"
)
return {
"user": user_resp.json(),
"orders": orders_resp.json()
}
async def fetch_multiple_users(user_ids):
async with asyncio.TaskGroup() as tg:
tasks = [tg.create_task(fetch_user_data(uid)) for uid in user_ids]
results = [task.result() for task in tasks]
print(f"Fetched {len(results)} users with orders")
for r in results:
print(f" {r['user']['email']}: {len(r['orders'])} orders")
return results
# Run
# results = asyncio.run(fetch_multiple_users([1, 2, 3, 4, 5]))
# Expected output:
# Fetched 5 users with orders
# user1@test.com: 3 orders
# user2@test.com: 1 orders
# user3@test.com: 0 orders
httpx Features for Testing
import httpx
# HTTP/2 support
client = httpx.Client(http2=True)
response = client.get("https://api.dodatech.com/v1/users")
print(f"HTTP version: {response.http_version}")
# Expected: HTTP/2
# Connection pooling
# httpx reuses connections automatically
# Configure pool size
limits = httpx.Limits(
max_connections=100,
max_keepalive_connections=20,
keepalive_expiry=30.0
)
client = httpx.Client(limits=limits)
# Event hooks for monitoring
def log_request(request):
print(f"Request: {request.method} {request.url}")
def log_response(response):
print(f"Response: {response.status_code} ({response.elapsed.total_seconds():.2f}s)")
client = httpx.Client(
event_hooks={
"request": [log_request],
"response": [log_response]
}
)
# Mock transport for tests
import httpx
def test_with_mock():
"""Use mock transport to avoid real HTTP calls."""
def handler(request):
return httpx.Response(
status_code=200,
json={"id": 1, "email": "test@test.com"}
)
client = httpx.Client(transport=httpx.MockTransport(handler))
response = client.get("https://api.test.com/users/1")
assert response.json()["email"] == "test@test.com"
aiohttp for API Testing
import aiohttp
import asyncio
import pytest
# Basic aiohttp test
@pytest.mark.asyncio
async def test_with_aiohttp():
async with aiohttp.ClientSession(
base_url="https://api.dodatech.com/v1"
) as session:
async with session.get("/users") as response:
assert response.status == 200
data = await response.json()
assert len(data) > 0
# Concurrent requests with aiohttp
async def fetch_all_pages(session, base_url, total_pages):
async def fetch_page(page):
async with session.get(f"{base_url}/users?page={page}") as resp:
return await resp.json()
tasks = [fetch_page(p) for p in range(1, total_pages + 1)]
results = await asyncio.gather(*tasks)
all_users = [user for page in results for user in page]
print(f"Total users across {total_pages} pages: {len(all_users)}")
return all_users
# async with aiohttp.ClientSession() as session:
# users = await fetch_all_pages(session, "https://api.dodatech.com/v1", 5)
WebSocket Testing
import asyncio
import httpx
# httpx WebSocket support
async def test_websocket():
async with httpx.AsyncClient() as client:
async with client.ws_connect(
"wss://api.dodatech.com/v1/ws/notifications",
headers={"Authorization": "Bearer token"}
) as websocket:
# Send message
await websocket.send_json({"type": "subscribe", "channel": "orders"})
# Receive messages
for _ in range(3):
message = await websocket.receive_json()
print(f"WS message: {message['type']}")
assert message["status"] == "ok"
# aiohttp WebSocket
async def test_websocket_aiohttp():
async with aiohttp.ClientSession() as session:
async with session.ws_connect(
"wss://api.dodatech.com/v1/ws",
headers={"Authorization": "Bearer token"}
) as ws:
await ws.send_json({"action": "ping"})
async for msg in ws:
if msg.type == aiohttp.WSMsgType.TEXT:
data = msg.json()
print(f"Received: {data}")
if data.get("type") == "pong":
break
Common Mistakes
1. Not Using Async Context Managers
async with httpx.AsyncClient() as client: properly closes connections. Forgetting async with leaks connections and causes test warnings about unclosed transports.
2. Mixing Sync and Async Without Care
Calling async code from sync tests requires asyncio.run() or pytest-asyncio. Mixing them without proper event loop management causes RuntimeError about conflicting event loops.
3. Not Setting Limits for Load Tests
Default connection limits are conservative. For load tests, explicitly set Limits(max_connections=1000) to allow concurrent requests without Connection Pool exhaustion.
4. Forgetting Response Body Await
In aiohttp, response.json() is a Coroutine. Forgetting await returns a coroutine object instead of data. In httpx, response.json() is synchronous.
5. Not Handling WebSocket Disconnects
WebSocket connections can drop. Always handle websocket.close() or ConnectionClosed exceptions. Implement reconnect logic for tests that need continuous WS connectivity.
Practice Questions
- What is the difference between httpx and requests?
- How do you make concurrent HTTP requests with httpx?
- How do you test WebSocket endpoints?
- What are httpx event hooks used for?
Answers:
- httpx supports async/await, HTTP/2, WebSockets, and modern connection pooling. requests is synchronous only, HTTP/1.1, with a simpler API. Use httpx for new projects, requests for legacy code.
- Use
asyncio.gather(*tasks)with multiple async HTTP calls. Create tasks withclient.get()inside async functions, gather them, and await all results concurrently. - Use httpx:
client.ws_connect(url)returns a WebSocket session withsend_json()andreceive_json(). Use aiohttp:session.ws_connect(url)withsend_json()and async iteration over incoming messages. - Event hooks allow custom callbacks on request/response lifecycle events. Use them for logging, timing, modifying requests, or injecting delays for testing.
Challenge: Build a test suite that uses async httpx for an e-commerce API: concurrent user + orders + products fetching, WebSocket notification testing, mock transport for offline testing, event hooks for latency measurement, and compare sync vs async performance.
FAQ
Mini Project
Build an async API test suite using httpx: async CRUD tests for 3 resources, concurrent data fetching with asyncio.gather, WebSocket notification testing, mock transport for offline mode, event hooks for request logging, performance comparison with sync requests, and JUnit XML reporting.
What's Next
Mock Servers — mock external API dependencies for isolated testing.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro