7
$\begingroup$

I am trying to match new product description with the existing ones. Product description looks like this: Panasonic DMC-FX07EB digital camera silver. These are steps to be performed:

  1. Tokenize description and recognize attributes: Panasonic => Brand, DMC-FX07EB => Model, etc.
  2. Get few candidates with similar features
  3. Get the best candidate.

I am having problem with the first step (1). In order to get 'Panasonic => Brand', DMC-FX07EB => Model, silver => color, I need to have index where each token of the product description correspond to certain attribute name (Brand, model, color, etc.) in the existing database. The problem is that in my database product descriptions are presented as one atomic attribute e.g. 'description' (no separated product attributes).

Basically I don't have training data, so I am trying to build index of all product attributes so I can build training data. So far I have attributes from bestbuy.com and semantics3.com APIs, but both sources lack most of attributes or contain irrelevant ones. Any suggestions for better APIs to get product attributes? Better approach to do this?

P.S. For every product there is a matched product description in the Database, which is as well in a form of one atomic attribute. I have checked this question on SO, it helped me and it seems we have same approach but I am still trying to get training data.

$\endgroup$
5
$\begingroup$

Left you a quick response on SO. The gist is that you can collect a lot of information from electronics shops and manufacturers' web sites, and lots you can annotate manually. If your goal is to only get training data, that's all you need:

My answer form the cross-post: "Having developed a commercial analyzer of this kind, I can tell you that there is no easy solution for this problem. But there are multiple shortcuts, especially if your domain is limited to cameras/electronics.

Firstly, you should look at more sites. Many have product brand annotated in the page (proper html annotations, bold font, all caps in the beginning of the name). Some sites have entire pages with brand selectors for search purposes. This way you can create a pretty good starter dictionary of brand names. Same with product line names and even with models. Alphanumeric models can be extracted in bulk by regular expressions and filtered pretty quickly.

There are plenty of other tricks, but I'll try to be brief. Just a piece of advice here: there is always a trade-off between manual work and algorithms. Always keep in mind that both approaches can be mixed and both have return-on-invested-time curves, which people tend to forget. If your goal is not to create an automatic algorithm to extract product brands and models, this problem should have limited time budget in your plan. You can realistically create a dictionary of 1000 brands in a day, and for decent performance on known data source of electronic goods (we are not talking Amazon here or are we?) a dictionary of 4000 brands may be all you need for your work. So do the math before you invest weeks into the latest neural network named entity recognizer."

$\endgroup$
  • $\begingroup$ You might instead reproduce your answer here, although ideally it wouldn't have been cross-posted as a question initially. $\endgroup$ – Sean Owen Dec 3 '14 at 13:52
  • $\begingroup$ Thank you. I guess I don't have much of a choice here. It's not just about cameras, basically parent category is, lets say, electronics with all sub-cats: cameras, smartphones, laptops etc. But so far there is no better solution than to build manually. Online catalogs would help me, but like I said, most of them contain more irrelevant info than relevant-at least in my case. $\endgroup$ – dzeno Dec 4 '14 at 8:33
  • $\begingroup$ Electronics in general is a huge category with long tail distribution. A lot would depend on your specific needs. For example, taking the "assorted" category in: if you do that, you will be swamped with tons of Chinese small name brands, and your best bet would be reliable feature recognition (memory size, color, domain keywords, etc.). If your target metric is purely numerical, "Panasonics" would suddenly constitute a shrinking minority. But if it's "major electronics", try compiling a brand dictionary from web sites first. $\endgroup$ – Alex Nevidomsky Dec 4 '14 at 10:02
  • 1
    $\begingroup$ Btw, I would be interested to see the results of your experiments - if you intend making them public. $\endgroup$ – Alex Nevidomsky Dec 4 '14 at 10:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.