Pandas CategoricalDtype Quick Fix
You will learn how to properly use CategoricalDtype in pandas to represent categorical data with fixed categories, saving memory and enabling ordered comparisons.
The Problem
Handling categorical data as raw strings or integers wastes memory and disables pandas categorical optimizations like grouping, sorting by custom order, and category-based encoding. This quick-fix guide covers the correct CategoricalDtype usage and common pitfalls.
The Wrong Way
import pandas as pd
s = pd.Series(["low", "medium", "high", "low", "high"])
print(s.memory_usage(deep=True))
Output:
235 # memory in bytes using Python strings
Each string is stored as a full Python object with no shared representation.
The Right Way
from pandas.api.types import CategoricalDtype
cat_type = CategoricalDtype(categories=["low", "medium", "high"], ordered=True)
s = pd.Series(["low", "medium", "high", "low", "high"], dtype=cat_type)
print(s.memory_usage(deep=True))
print(s)
Output:
180 # significantly less memory
0 low
1 medium
2 high
3 low
4 high
dtype: category
Categories (3, object): ['low' < 'medium' < 'high']
Step-by-Step Fix
1. Define the category dtype
Always specify the full list of categories and whether ordering matters.
2. Apply during Series creation or with .astype()
s = pd.Series(["a", "b", "c"]).astype(CategoricalDtype(ordered=True))
3. Verify the result
print(s.dtype)
print(s.cat.categories)
Prevention Tips
- Use
CategoricalDtypeinstead of raw strings for columns with few unique values - Always set
ordered=Truewhen categories have a natural order - Specify categories explicitly to guard against unseen test values
- Use
.cat.codesfor integer encoding for ML models
Common Mistakes
- Forgetting to set
ordered=Truewhen order matters -- sorting defaults to alphabetical - Using
pd.Categorical()without specifying categories -- categories are inferred from data - Not converting before grouping -- groupby on strings is slower than on categoricals
These mistakes appear frequently in real-world pandas code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.
FAQ
Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro. DodaTech tools integrate seamlessly with pandas for enhanced productivity and security.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro