Remove Duplicates from a List in Python

To remove duplicates from a list in Python, you can use methods like set(), dict.fromkeys(), list comprehension, or a loop to filter out duplicate elements. These techniques ensure that only unique elements remain in the list while preserving or discarding order as required based on the technique you are choosing. In this tutorial, we will go through each of these techniques with examples.

Examples

1. Removing Duplicates Using set()

The simplest way to remove duplicates from a list is by converting it into a set and then back into a list. However, this method does not preserve the original order of elements.

</>
Copy
# Original list with duplicates
numbers = [1, 2, 2, 3, 4, 4, 5, 6, 6, 7]

# Removing duplicates using set()
unique_numbers = list(set(numbers))

# Printing the updated list
print("List after removing duplicates:", unique_numbers)

Explanation:

  • The original list numbers contains duplicate values.
  • The set() function removes duplicates since sets only store unique values.
  • We convert the set back into a list using list(set(numbers)).

Output:

List after removing duplicates: [1, 2, 3, 4, 5, 6, 7]

2. Removing Duplicates While Preserving Order Using dict.fromkeys()

The dict.fromkeys() method removes duplicates while preserving the original order of elements.

</>
Copy
# Original list with duplicates
words = ["apple", "banana", "apple", "orange", "banana", "grape"]

# Removing duplicates while preserving order
unique_words = list(dict.fromkeys(words))

# Printing the updated list
print("List after removing duplicates:", unique_words)

Explanation:

  • The original list words contains duplicate fruit names.
  • dict.fromkeys(words) creates a dictionary where the keys are the list elements (removing duplicates automatically).
  • We convert the dictionary back into a list using list() to preserve order.

Output:

List after removing duplicates: ['apple', 'banana', 'orange', 'grape']

3. Removing Duplicates Using a Loop

If you want full control over how duplicates are handled while preserving order, you can use a loop and a set to track seen elements.

</>
Copy
# Original list with duplicates
colors = ["red", "blue", "red", "green", "blue", "yellow"]

# Using a loop to remove duplicates while preserving order
unique_colors = []
seen = set()

for color in colors:
    if color not in seen:
        unique_colors.append(color)
        seen.add(color)

# Printing the updated list
print("List after removing duplicates:", unique_colors)

Explanation:

  • The original list colors contains duplicate color names.
  • A new empty list unique_colors is created to store unique elements.
  • A set called seen keeps track of elements that have already been added.
  • Using a for loop, we check if an element is in seen. If not, we add it to unique_colors and update seen.

Output:

List after removing duplicates: ['red', 'blue', 'green', 'yellow']

4. Removing Duplicates Using List Comprehension

List comprehension offers a compact way to remove duplicates while maintaining order.

</>
Copy
# Original list with duplicates
items = [10, 20, 10, 30, 40, 30, 50]

# Removing duplicates using list comprehension
seen = set()
unique_items = [x for x in items if not (x in seen or seen.add(x))]

# Printing the updated list
print("List after removing duplicates:", unique_items)

Explanation:

  • The original list items contains duplicate integers.
  • The seen set tracks encountered values.
  • List comprehension iterates over items, adding an element to unique_items only if it hasn’t been encountered before.

Output:

List after removing duplicates: [10, 20, 30, 40, 50]

Conclusion

There are multiple ways to remove duplicates from a list in Python:

  1. set(): Quick and simple, but does not preserve order.
  2. dict.fromkeys(): Removes duplicates while preserving order.
  3. Loop with a Set: Manually tracks seen elements and preserves order.
  4. List Comprehension: A compact and Pythonic way to filter unique elements.