Remove Duplicate Dictionaries from Nested Dictionary - Python

We are given a nested dictionary we need to remove the duplicate dictionaries from the nested dictionary. For example we are given a nested dictionary d = {'key1': [{'a': 1}, {'b': 2}, {'a': 1}], 'key2': [{'x': 3}, {'y': 4}]} we need to remove the duplicate dictionary from this dictionary so output should be {'key1': [{'a': 1}, {'b': 2}], 'key2': [{'y': 4}, {'x': 3}]}. We can use sets , list comprehension for this.

Using a Set

A simple way is to convert list of dictionaries to a set of frozen sets (since sets cannot contain duplicate elements) and then back to a list.

{'key1': [{'b': 2}, {'a': 1}], 'key2': [{'y': 4}, {'x': 3}]}

Explanation:

• Convert dictionaries to frozensets to remove duplicates.
• Convert frozensets back to dictionaries to restore structure.

Using a Loop and a Set

We can manually track seen dictionaries using a set to avoid duplicates.

{'key1': [{'a': 1}, {'b': 2}], 'key2': [{'x': 3}, {'y': 4}]}

Explanation:

• Track unique dictionaries for each list of dictionaries, convert each dictionary to a frozenset and use a set to track previously seen dictionaries ensuring only unique dictionaries are kept.
• Update dictionary after processing each list update the dictionary with list of unique dictionaries removing any duplicates.

Using List Comprehension and a Set

Code uses list comprehension and a set to remove duplicates from lists of dictionaries. It converts each dictionary to a frozenset to ensure uniqueness then iterates through list adding only unique dictionaries to a new list which is then used to update the original dictionary.

{'key1': [{'a': 1}, {'b': 2}], 'key2': [{'x': 3}, {'y': 4}]}

Explanation:

• Loop through dictionary Iterates through the dictionary, processing each key and its associated list of dictionaries.
• List comprehension for each dictionary in the list it converts the dictionary to a frozenset and checks if it's already in list of seen dictionaries. Only unique dictionaries are appended to list.

Using pandas

If we are working with large datasets we can use pandas to convert dictionaries into a DataFrame and remove duplicates easily.

{'key1': [{'a': 1.0, 'b': nan}, {'a': nan, 'b': 2.0}], 'key2': [{'x': 3.0, 'y': nan}, {'x': nan, 'y': 4.0}]}

Explanation:

• Convert to DataFrame and remove duplicates each list of dictionaries is converted to a pandas DataFrame where duplicates are removed using the drop_duplicates() method.
• Convert back to list of dictionaries dataFrame is converted back into a list of dictionaries with the to_dict(orient='records') method, updating the original dictionary.