We have a ruby-on-rails platform (w/ postgreSQL db) for people to upload various products to trade. Of course, many of these products listed are the same, while they are described differently by the consumer (either through spelling, case etc.) "lots of duplicates"
For the purposes of analytics and a better UX, we're aiming to create an evolving "master product list", or "whitelist", if you will, that will have users select from an existing list of products they are uploading, OR request to add a new one. We also plan to enrich each product entry with additional information from the web, that would be tied to the "master product".
Here are some methods we're proposing to solve this problem:
A) Take all the "items" listed in the website (~90,000), de-dupe as much as possible by running select "distinct" queries (while maintaining a key-map back to original data by generating an array of item keys from each distinct listing in a group-by.)
A1) Running this data through mechanical turk, and asking each turk user to list data in a uniform format.
A2) Running each product entry through the Amazon products API and asking the user to identify a match.
A3) A better method?