Pandas Find Duplicate Rows

Most simple way to find duplicate rows in DataFrame is by using the duplicated() method. This method returns a boolean Series indicating whether each row is a duplicate of a previous row.

 Name Age Gender Salary
4 John 25 Male 50000

In addition to the this method, there are several other approaches to find out the Duplicate rows in Pandas.

1. Identifying Duplicates Based on Specific Columns

Sometimes, you may only want to check for duplicates in specific columns, rather than the entire row. we can use duplicated() with the subset parameter to check for duplicates in specific columns.

The subset parameter allows you to specify which columns to consider when looking for duplicates. In this case, we're checking for duplicates based on the Name and Age columns.

 Name Age Gender Salary
4 John 25 Male 50000

2. Removing Duplicate Rows Using drop duplicates

Once duplicates are identified, you can remove them using the drop_duplicates() method. This method returns a new DataFrame with the duplicate rows removed.

The drop_duplicates() method removes all rows that are identical to a previous row. By default, it keeps the first occurrence of each row and drops subsequent duplicates.

 Name Age Gender Salary
0 John 25 Male 50000
1 Alice 30 Female 55000
2 Bob 22 Male 40000
3 Eve 35 Female 70000
5 Charlie 28 Male 48000

Here are some key takeaways:

• Identify duplicates using duplicated()
• Find duplicates based on specific columns
• Remove duplicate rows with drop_duplicates()