Python Functools — Higher-Order Functions and Decorators
In this tutorial, you will learn about Python Functools. We cover key concepts, practical examples, and best practices to help you master this topic.
The functools module provides higher-order functions that work with other functions. It's a small but mighty module that every Python developer should know. From Caching to partial application to function overloading, functools makes advanced Functional Programming patterns accessible.
In this tutorial, you'll learn the most useful functools utilities. DodaTech uses functools.lru_cache to speed up repetitive threat signature lookups, and partial to pre-configure database connection parameters.
What You'll Learn
- lru_cache: automatic memoization for expensive functions
- partial: freeze function arguments
- reduce: cumulative operations on iterables
- wraps: preserving metadata in decorators
- singledispatch: function overloading by type
- cmp_to_key: support for old-style comparison functions
lru_cache — Automatic Memoization
When you have an expensive function called repeatedly with the same arguments, lru_cache saves results:
from functools import lru_cache
import time
@lru_cache(maxsize=128)
def fibonacci(n):
"""Return the nth Fibonacci number."""
if n < 2:
return n
return fibonacci(n - 1) + fibonacci(n - 2)
# First call: actually computes
start = time.time()
result = fibonacci(35)
print(f"fib(35) = {result}, took {time.time() - start:.4f}s")
# Second call: cached result
start = time.time()
result = fibonacci(35)
print(f"fib(35) = {result}, took {time.time() - start:.6f}s (cached)")
# Check cache stats
print(fibonacci.cache_info())
# CacheInfo(hits=33, misses=36, maxsize=128, currsize=36)
Expected output:
fib(35) = 9227465, took 0.2345s
fib(35) = 9227465, took 0.000001s (cached)
CacheInfo(hits=33, misses=36, maxsize=128, currsize=36)
partial — Pre-Fill Function Arguments
from functools import partial
# A function with many parameters
def power(base, exponent):
return base ** exponent
# Create specialized versions
square = partial(power, exponent=2)
cube = partial(power, exponent=3)
print(square(5)) # 25
print(cube(5)) # 125
print(power(5, 4)) # 625 (original still works)
# Real-world: database connection
def connect(host, port, db, user, password, timeout=30):
print(f"Connecting to {host}:{port}/{db} as {user}")
# Pre-configure for production
prod_db = partial(connect, host="prod.example.com", port=5432,
db="prod", user="admin", timeout=60)
# Now call with just password
prod_db(password="secret123")
reduce — Cumulative Operations
from functools import reduce
# Sum all numbers (like sum(), but more flexible)
numbers = [1, 2, 3, 4, 5]
total = reduce(lambda a, b: a + b, numbers)
print(total) # 15
# Find the maximum
max_val = reduce(lambda a, b: a if a > b else b, numbers)
print(max_val) # 5
# Compute factorial
factorial = reduce(lambda a, b: a * b, range(1, 6))
print(factorial) # 120 (5!)
# With initial value
product = reduce(lambda a, b: a * b, numbers, 10)
print(product) # 1200 = 10 * 1 * 2 * 3 * 4 * 5
wraps — Preserve Decorator Metadata
from functools import wraps
# Without wraps — metadata is lost
def bad_decorator(func):
def wrapper(*args, **kwargs):
"""Wrapper docstring"""
return func(*args, **kwargs)
return wrapper
@bad_decorator
def hello():
"""Say hello."""
print("Hello!")
print(hello.__name__) # "wrapper" (WRONG!)
print(hello.__doc__) # "Wrapper docstring" (WRONG!)
# With wraps — metadata is preserved
def good_decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
"""Wrapper docstring"""
return func(*args, **kwargs)
return wrapper
@good_decorator
def hello():
"""Say hello."""
print("Hello!")
print(hello.__name__) # "hello" (CORRECT!)
print(hello.__doc__) # "Say hello." (CORRECT!)
Common Pitfalls
Pitfall 1: lru_cache on Unhashable Arguments
@lru_cache
def process_list(items): # TypeError: unhashable type: 'list'
return sum(items)
# Fix: convert to tuple
@lru_cache
def process_tuple(items):
return sum(items)
process_tuple((1, 2, 3)) # Works
Pitfall 2: lru_cache Memory Issues
For functions with many unique arguments, the cache grows unbounded. Use maxsize or consider an alternative:
@lru_cache(maxsize=256) # Limit cache size
def expensive_api_call(url):
return fetch(url)
Practice Questions
Use partial to create a list of formatting functions: euro(amount), dollar(amount), yen(amount) that format as "xx.xx EUR", "$xx.xx", "xx JPY".
Use lru_cache to optimize a recursive function that computes the Collatz sequence length for numbers up to 1 million.
Use reduce to implement a pipeline: a list of functions [f1, f2, f3] applied left-to-right to an initial value.
Write a decorator with @wraps that logs function arguments and return value.
Use singledispatch to create a function format_output(x) that handles str, int, float, and list differently.
Challenge: Memoized Fibonacci with TTL
Extend lru_cache to support time-to-live (TTL) expiration. Create a decorator @ttl_cache(maxsize=128, ttl=60) that caches results for at most 60 seconds before re-computing. This is useful for caching API responses that change over time, similar to how DodaTech's threat intelligence cache refreshes every 5 minutes.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro