How to Remove Duplicates from a Python List

By | October 25, 2020
How to Remove Duplicates from a Python List

Python list is a very flexible and widely used Python data structure. Python does not support static Arrays like other high-level programming languages, instead, it has List which is an alternative for Python array.

A Python list can contain multiple data values, and to access an individual value we can use the value corresponding index number. In this Python tutorial article, we will learn the different methods on how to remove duplicates from a list. A list can have multiple elements and those elements can be similar too, and in a Python interview, you may encounter this question on removing duplicate elements from a list.

Vamware

Prerequisites for this article

Here we have provided 4 different techniques you can use to remove duplicate elements from a list.

  1. Remove Duplicate elements using Naïve Method
  2. Remove Duplicate elements using set()
  3. Remove Duplicate elements using list comprehension and enumerate() function
  4. Remove Duplicate elements using dictionary .formkeys() method

Remove Duplicate Elements from a List using Naïve Method

In the naïve method, we will create a procedural algorithm to remove duplicate elements from a list.

In this approach first, we will create a new empty list and traverse through every element from our existing list. Inside the loop, we will check for each element if it is present in the new list, if yes then we will move further if not we will insert that element in our new list. With this when the loop end, we will have a list with no duplicate values.

Example

exis_list = [ 1,2,2,3,4,5,6,7,5,4,2,3,7,8]
new_list = []
for i in exis_list:
    if not i in new_list:
        new_list.append(i)

print("The old list is:", exis_list)
print("The new list is:", new_list)

Output

The old list is: [1, 2, 2, 3, 4, 5, 6, 7, 5, 4, 2, 3, 7, 8]
The new list is: [1, 2, 3, 4, 5, 6, 7, 8]

Remove Duplicate Elements from a List using set()

A Python set is an unordered and immutable Python data structure. A python set does not store duplicate values. And with the help of set() method, we can convert a list object to the set object. If we convert the list into a set, all the duplicate elements from the list get omitted.

Example

exis_list = [ 1,2,2,3,4,5,6,7,5,4,2,3,7,8]
new_set = set(exis_list)
new_list = list(new_set)

print("The old list is:", exis_list)
print("The new list is:", new_list)

Output

The old list is: [1, 2, 2, 3, 4, 5, 6, 7, 5, 4, 2, 3, 7, 8]
The new list is: [1, 2, 3, 4, 5, 6, 7, 8]

Remove Duplicate Elements from a List using list comprehension and enumerate()

The enumerate function can accept a List object and return an enumerate object which contains a pair tuple with index numbers and corresponding list value.

Using the list comprehension, we can create a new list that looks for the elements which are occurring the first time and add them.

exis_list = [ 1,2,2,3,4,5,6,7,5,4,2,3,7,8]
new_list = [ value for index,value in enumerate(exis_list) if value not in exis_list[:index]]

print("The old list is:", exis_list)
print("The new list is:", new_list)

Output

The old list is: [1, 2, 2, 3, 4, 5, 6, 7, 5, 4, 2, 3, 7, 8]
The new list is: [1, 2, 3, 4, 5, 6, 7, 8] 

Remove Duplicate Elements from a List using .fromkeys()

The .formkeys() is a dictionary method which is used to convert the list values to dictionary keys. A dictionary can only have immutable and unique keys. And the .formkeys() accept a 1D list and create a dictionary with list values as keys, and set their values to None. After creating the dictionary using dict.fromkeys() method we if we can change back the dictionary to the list using list() method.

Example

exis_list = [ 1,2,2,3,4,5,6,7,5,4,2,3,7,8]
new_dict = dict.fromkeys(exis_list)
new_list = list(new_dict)

print("The old list is:", exis_list)
print("The new list is:", new_list)

Output

The old list is: [1, 2, 2, 3, 4, 5, 6, 7, 5, 4, 2, 3, 7, 8]
The new list is: [1, 2, 3, 4, 5, 6, 7, 8]

Conclusion

With this, we have reached the end of our tutorial on How to remove duplicate values from a list. For most of the cases, you will be using the Naïve ad .fromkeys() methods. But using the set() method to create a unique list is a very clean and fastest method.

Leave a Reply

Your email address will not be published. Required fields are marked *