I am trying to clean a set of e-commerce data. Data consists of products which has multiple images, a title and a description. Some of these products has duplicates and I need to de-duplicate the set. I have worked on NN's and Genetic Algorithms back in college but that's it.

What I need is a starting point to dive into. What methods/techniques should I use, what should I learn first?


1 Answer 1


As a starting point chapter 3 of Mining Massive Datasets (MMDS) offers a very good introduction on document similarity. You can also look here. For near-duplicate image detection look at this question on StackOverflow.

For the text data, first you have to convert your strings to a vector of features, this features could be tf-idf vectors of all the tokens (words) that appear in the database, could also be a vector of n-grams. For a discussion of semantic and syntactic features you can look here.

Given the vectors for each product on your database you would have to compute the similarity among all pair of products. Then the most common similarities measures are cosine similarity and Jaccard similarity.

The brute force approach is O(n^2), where is the number of products in the database so this could be unfeasible. A common technique to deal with this issue is to use fingerprinting (hashing) together with Locality Sensitive Hashing (LSH), both techniques are discussed in chapter 3 of MMDS.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.