3
$\begingroup$

We have a ruby-on-rails platform (w/ postgreSQL db) for people to upload various products to trade. Of course, many of these products listed are the same, while they are described differently by the consumer (either through spelling, case etc.) "lots of duplicates"

For the purposes of analytics and a better UX, we're aiming to create an evolving "master product list", or "whitelist", if you will, that will have users select from an existing list of products they are uploading, OR request to add a new one. We also plan to enrich each product entry with additional information from the web, that would be tied to the "master product".

Here are some methods we're proposing to solve this problem:

A) Take all the "items" listed in the website (~90,000), de-dupe as much as possible by running select "distinct" queries (while maintaining a key-map back to original data by generating an array of item keys from each distinct listing in a group-by.)

THEN

A1) Running this data through mechanical turk, and asking each turk user to list data in a uniform format.

OR

A2) Running each product entry through the Amazon products API and asking the user to identify a match.

or

A3) A better method?

$\endgroup$
  • $\begingroup$ I'm not sure this is directly related to data science, but just a broad design approach question? $\endgroup$ – Sean Owen Nov 13 '14 at 4:46
  • $\begingroup$ Yes - I agree it's a rather broad design approach question. If you think I should take down and put elsewhere, happy to do so. $\endgroup$ – Nick Lashinsky Nov 14 '14 at 6:22
  • $\begingroup$ ETL tools have fuzzy logic built into them that can match like terms. Also postgresql 9.0 (and greater) has Levenshtein algorithm in the fuzzystrmatch function $\endgroup$ – Adrian Torrie Jul 27 '15 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.