4
$\begingroup$

We have a ruby-on-rails platform (w/ postgreSQL db) for people to upload various products to trade. Of course, many of these products listed are the same, while they are described differently by the consumer (either through spelling, case etc.) "lots of duplicates"

For the purposes of analytics and a better UX, we're aiming to create an evolving "master product list", or "whitelist", if you will, that will have users select from an existing list of products they are uploading, OR request to add a new one. We also plan to enrich each product entry with additional information from the web, that would be tied to the "master product".

Here are some methods we're proposing to solve this problem:

A) Take all the "items" listed in the website (~90,000), de-dupe as much as possible by running select "distinct" queries (while maintaining a key-map back to original data by generating an array of item keys from each distinct listing in a group-by.)

THEN

A1) Running this data through mechanical turk, and asking each turk user to list data in a uniform format.

OR

A2) Running each product entry through the Amazon products API and asking the user to identify a match.

or

A3) A better method?

Improve this question
$\endgroup$
3
  • $\begingroup$ I'm not sure this is directly related to data science, but just a broad design approach question? $\endgroup$
    – Sean Owen
    Nov 13, 2014 at 4:46
  • $\begingroup$ Yes - I agree it's a rather broad design approach question. If you think I should take down and put elsewhere, happy to do so. $\endgroup$
    – Nick Lashinsky
    Nov 14, 2014 at 6:22
  • $\begingroup$ ETL tools have fuzzy logic built into them that can match like terms. Also postgresql 9.0 (and greater) has Levenshtein algorithm in the fuzzystrmatch function $\endgroup$
    – Adrian Torrie
    Jul 27, 2015 at 10:52

0

Reset to default

Browse other questions tagged or ask your own question.