0
$\begingroup$

I am new to ML and still learning it.

My problem is to identify duplicate products. I have a dataset containing product details such as name, colour, size, description, features etc (there are roughly 70 columns).

I need to remove duplicate products.

I just completed some of the supervised ML model(classification and regression) and unsupervised clustering(K means and HC). I am also on the way of learning w2v and d2v.

But due to time constrain, I need to deliver a solution to the above problem statement. I am unsure as to how to proceed.

Any help and guidance would be appreciated

$\endgroup$
  • $\begingroup$ It doesn't look like an ML problem at all. Can't you simply find the duplicates by checking if all their features match? $\endgroup$ – Valentin Calomme May 20 at 13:42
0
$\begingroup$

This problem is called record linkage, there are various techniques which can be used, usually involving some distance measure between record and/or approximate string matching between string fields.

Fyi it's a quite complex problem, especially if quality deduplication is expected and the volume of data is high.

| improve this answer | |
$\endgroup$
0
$\begingroup$

You can do a K Means Clustering to see cluster your products and see if some products is situated very closely. (In the same cluster). Then you can say that products in the same cluster are similar. But you have to find the optimal k value of clusters.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.