Skip to content

Pandas drop_duplicates Not Removing Fix

DodaTech Updated 2026-06-24 2 min read

In this tutorial, you'll learn about Pandas drop_duplicates Not Removing Fix. We cover key concepts, practical examples, and best practices.

Pandas drop_duplicates() fails to remove all duplicates when trailing whitespace, case differences, or NaN values make rows appear unique when they are not.

The Wrong Way

import pandas as pd

df = pd.DataFrame({"Name": ["Alice", "Alice ", "BOB", "Bob"],
                   "Score": [85, 85, 92, 92]})
df = df.drop_duplicates()
print(df)

Output:

    Name  Score
0  Alice     85
1  Alice     85
2    BOB     92
3    Bob     92

All rows appear unique because of trailing whitespace in "Alice " and case differences in "BOB" vs "Bob".

The Right Way

import pandas as pd

df = pd.DataFrame({"Name": ["Alice", "Alice ", "BOB", "Bob"],
                   "Score": [85, 85, 92, 92]})
df["Name"] = df["Name"].str.strip().str.lower()
df = df.drop_duplicates()
print(df)

Output:

    Name  Score
0  alice     85
2    bob     92

Strip whitespace and normalize case before calling drop_duplicates.

Step-by-Step Fix

1. Check for hidden whitespace

df["Name"] = df["Name"].str.strip()

2. Normalize case before dedup

df["Name"] = df["Name"].str.lower()

3. Use subset to compare only specific columns

df = df.drop_duplicates(subset=["Name"])

4. Keep the last occurrence instead of first

df = df.drop_duplicates(keep="last")

5. Combine with sorting for deterministic results

df = df.sort_values("Date").drop_duplicates(subset="Name", keep="last")

Prevention Tips

  • Strip whitespace from string columns before deduplication.
  • Normalize case with .str.lower() or .str.upper().
  • Use subset parameter to specify which columns define duplicates.
  • Use keep='last' to keep the most recent occurrence.
  • Use df.duplicated() to inspect duplicates before dropping them.

Common Mistakes with drop duplicates

  1. Placing the wildcard pattern first in case expressions, making all subsequent patterns unreachable
  2. Using head and tail instead of pattern matching, causing runtime errors on empty lists
  3. Forgetting that lazy evaluation defers computation until the value is forced, causing space leaks with unevaluated thunks

These mistakes appear frequently in real-world PANDAS code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ

### Why does drop_duplicates keep all rows?

Hidden whitespace, different case, NaN values, or floating-point precision differences can make seemingly identical rows appear distinct. Use .str.strip() and normalize before dedup.

What is the difference between keep='first' and keep='last'?

keep='first' (default) keeps the first occurrence and drops subsequent duplicates. keep='last' keeps the last occurrence. Use keep=False to drop all rows that have any duplicates.

How do I find duplicate rows without dropping them?

Use df[df.duplicated()] to see only the duplicated rows, or df.duplicated(keep=False) to see all rows that participate in duplicates.

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.

Built by the developers of DodaTech

Doda Browser, DodaZIP & Durga Antivirus Pro