python - pandas.series.drop_duplicates() cannot delete all duplicates

Question

>>> info
0                       (dataset, license, sources, weight)
1                       (dataset, license, sources, weight)
2                       (dataset, license, sources, weight)
3                       (dataset, license, sources, weight)
4                       (dataset, license, sources, weight)
                                ...                        
491877    (dataset, license, sources, surfaceEnd, surfac...
491878    (dataset, license, sources, surfaceEnd, surfac...
491879    (dataset, license, sources, surfaceEnd, surfac...
491880    (dataset, license, sources, surfaceEnd, surfac...
491881    (dataset, license, sources, surfaceEnd, surfac...
Name: edge_info, Length: 491882, dtype: object

>>> info.drop_duplicates()
0                   (dataset, license, sources, weight)
1                   (dataset, license, sources, weight)
70    (dataset, license, sources, surfaceEnd, surfac...
71    (dataset, license, sources, surfaceEnd, surfac...
Name: edge_info, dtype: object

>>> info.iloc[0]==info.iloc[1]
True
>>> info.iloc[0]==info.iloc[2]
True
>>> info.iloc[0]
dict_keys(['dataset', 'license', 'sources', 'weight'])
>>> 

The above commands require a series object to drop duplicate items.

However, the results seem that still have duplicate value as show above.

The first row info.iloc[0] and the second row info.iloc[1] of info is equal, but the drop_duplicates() function do not remove the second item.

Does any know the reason for the results?

Answer 1

The reason the first and second row is equal that you are still checking the original dataframe, not the output you removed duplicates from. Most functions (but not all, always check the documentation) in pandas that modify DataFrame/Series, they don't change the original data by default instead they just return the modified data.

To resolve your issue you have two options:

1. Assign the modified data to the variable::
   info = info.drop_duplicates()

2. Set parameter inplace=True(by default this is False).This will change the dataframe object in place and will return None.

   info.drop_duplicates(inplace=True)

Here are some useful links about both methods:

• In pandas, is inplace = True considered harmful, or not?

• Explaining the Inplace Parameter for Beginners