How to Remove Duplicate Values from JS Array

Last updated on December 27th, 2024 at 04:20 am

Sometimes JavaScript arrays contain duplicate values. Web developers need to remove duplicate values from JS arrays to obtain unique values, and do further processing. This is commonly required in data analysis. There are several ways to remove duplicates in JavaScript array. Some of them use plain JavScript while some use third-party JS library like jQuery. In this article, we will learn how to do this.

How to Remove Duplicate Values from JS Array

Here are the different ways to remove duplicate values from JS array. Let us say you have the following JavaScript array with duplicates.

var data = [1, 2, 2, 3, 4, 4, 5]

Now let us look at some of the commonly used methods to remove duplicates. Most of them can be done using plain JavaScript. You may also use popular JS libraries such as jQuery.

1. Using Set Constructor & Spread

JavaScript provides a Set() function that takes in an object as argument, and returns an object with unique values. In other words, it converts an object with duplicate values into a set with unique values. So we pass the original array to Set() function to remove duplicates.

new Set(data)

But Set() function returns a set of values, instead of array. So we pass the result of Set() function into an array constructor, to convert the result back into an array.

var new_data = [...new Set(data)]
console.log(new_data);

Here is the output array without duplicates.

[ 1, 2, 3, 4, 5 ]

2. Using Filter() and IndexOf() functions

Every JavaScript array supports filter() function that evaluates a specific condition for each element of the array, and creates a new array of all the elements that pass the condition. You can easily use this function along with other JavaScript functions to remove duplicates.

Here is a simple example to use it with indexOf() function. We will use filter() function to iterate over all elements of JS array. In each iteration, we will call the indexOf() function of the element. IndexOf() function will always return the position of the first occurrence of the element. So it will return the same index for original as well as duplicate items. If the result of indexOf() function is equal to the actual index of the element, it means that element is the first occurrence, else it is a duplicate and can be skipped. If an element is the first occurrence, it is added to the new array without duplicates.

var new_data = data.filter(function(item, pos) {
return data.indexOf(item) == pos;
});

console.log(new_data); // output is [ 1, 2, 3, 4, 5 ]

3. Using Hashtables

By definition, a hash table is a unique collection of key-value pairs with unique key values. For our purpose, we loop through the JS array, and enter whether the item has already been entered in hash table or not. If the element has already been encountered, we enter true for it, else we enter false for it. The filter() function automatically returns only those items for which we enter true value in hash table.

var seen={};

var new_data = data.filter(function(item) {
return seen.hasOwnProperty(item) ? false : (seen[item] = true);
});

console.log(new_data);

This is a very fast method that runs in linear time. However, it has only two problems. Since hash keys need to be strings, if two elements are similar, one being a number and the other being a string, then it will term them to be duplicates. For example, 1 and “1” are treated as duplicates of each other. Secondly, it also considers all objects as equal. For example, {abc:1} and {abc:2} are considered to be equal. If they do not apply to your data, then you can surely use it.

4. Using sort()

If your array contains simple data types such as numbers and strings, but not any objects, then you can first sort this array using sort() function. Then you can loop through the array items and remove those items that are equal to preceding element.

var new_data =  data.sort().filter(function(item, pos, ary) {
return !pos || item != ary[pos - 1];
});

console.log(new_data); // output is [ 1, 2, 3, 4, 5 ]

This is a very useful method if your array consists of numbers or strings that are sortable.

5. Using forEach() with array.includes()

In this approach, we first create an empty array to store unique items. Then we use forEach() function to loop through the original array, and in each iteration, we check if the element is already present in the new array consisting of original elements.

var new_data = [];
data.forEach(element => {
if (!new_data.includes(element)) {
new_data.push(element);
}
});

console.log(new_data); // output is [ 1, 2, 3, 4, 5 ]

6. Using reduce() with array.includes()

Reduce() function allows you to run a reducer callback function on every element of an array. In each iteration, it performs an operation using the present element and result of previous iteration. Lastly, you get a single value, which is the array of unique items in our case.

var new_data = data.reduce(function (acc, curr) {
if (!acc.includes(curr)) {
acc.push(curr);
}

return acc;
}, []);

console.log(new_data); // [ 1, 2, 3, 4, 5 ]

In the above code, our reducer function starts with an empty array. In each iteration, it checks if the item is present in this array, using includes() function. If not, then it adds the element to the array, using push() function. Lastly, it returns the array of unique items as result.

7. Using jQuery

If you use popular third-party JS library such as jQuery, then you can easily remove duplicates from arrays. Here is an example to illustrate it.

var new_data = [];

$.each(data, function(i, el){
if($.inArray(el, new_data) === -1) new_data.push(el);
});

console.log(new_data); // output is [ 1, 2, 3, 4, 5 ]

In the above code, we first create an empty array new_data to store unique elements. Then we use each() function to iterate through the original data array. In each iteration, we use inArray() function to check if the element is present in the new_data array. If not, then we use push() function to push the element into new array. inArray() returns -1, if the element is not present in the given array.

8. Adding prototype function

If you need to frequently remove duplicates from JavaScript arrays, then it is better to create a simple prototype function that can be called on every array on your website’s pages. You can do this using the prototype constructor. Within this function, you can add any of the above mentioned codes to remove duplicates. For our purpose, we will use the set() function used in solution #1 above.

Array.prototype.unique = function(){
return Array.from(new Set(this));
}

In the above code, we define a prototype function called unique() that can be called on all arrays on your web page. It basically calls Set() constructor on this array, converts the result into list and returns it.

You can call it as follows.

var data = [1, 2, 2, 3, 4, 4, 5];
console.log(data.unique()); // output is [1, 2, 3, 4, 5]

Conclusion

In this article, we have learnt how to remove duplicate values from JavaScript arrays. You can use any of these methods as per your requirement. Most of the solutions work well for all data types, such as numbers, strings. Among them, using set() function with spread operator is one of the most commonly used solutions. If you want to remove only specific duplicates, based on some condition, then you can use filter() function. It allows you to specify custom conditions to remove only some duplicates instead of all duplicates. As shown above, you can use various functions along with filter() function for this purpose. If your array contains objects, then it is advisable to use filter() function since they allow you to specifically examine each object and decide whether it is a duplicate or not.

FAQ

1. Can these solutions be used for mixed data arrays?

Yes. These methods work well whether your array contains numbers, text, or alphanumeric strings. They may not work if your array items contain objects. In this case, you will need to customize your filter() function to compare if two objects are same or not.

2. Do I need to use JavaScript Library to remove array duplicates?

No. You can easily remove duplicates using plain JavaScript. If you are using third-party libraries like jQuery, then you can also use them as required.

3. Can these solutions be used for large arrays?

Yes. The solutions that use Set() or filter() functions can be used for large arrays also.

Also read:

Difference Between Var and Let in JavaScript
How to Add Table Row in jQuery
How to Disable/Enable Input in jQuery

Leave a Reply

Your email address will not be published. Required fields are marked *