Remove Duplicates from a List in Python

Remove Duplicates from a List in Python

Cleaning a list of duplicate values is a frequent data processing task. Python offers several clean approaches depending on whether order matters.

Fastest Method: Convert to Set (Order Not Preserved)

my_list = [1, 2, 2, 3, 4, 4, 5]
unique_list = list(set(my_list))

print(unique_list)  # Output: [1, 2, 3, 4, 5] (order may vary)

Preserve Order: Using dict.fromkeys()

my_list = [1, 2, 2, 3, 4, 4, 5]
unique_list = list(dict.fromkeys(my_list))

print(unique_list)  # Output: [1, 2, 3, 4, 5]

Preserve Order: Using a Loop

def remove_duplicates(lst):
    seen = set()
    result = []
    for item in lst:
        if item not in seen:
            seen.add(item)
            result.append(item)
    return result

print(remove_duplicates([1, 2, 2, 3, 4, 4, 5]))  # Output: [1, 2, 3, 4, 5]

Remove Duplicates from a List of Strings

words = ["apple", "banana", "apple", "cherry", "banana"]
unique_words = list(dict.fromkeys(words))
print(unique_words)  # Output: ['apple', 'banana', 'cherry']

Key Takeaway

Use list(set(lst)) when order does not matter. Use dict.fromkeys() to preserve insertion order — it is clean, readable, and Pythonic.