Attributes extraction from unstructured product descriptions

Question

I am trying to match new product description with the existing ones. Product description looks like this: Panasonic DMC-FX07EB digital camera silver. These are steps to be performed:

1. Tokenize description and recognize attributes: Panasonic => Brand, DMC-FX07EB => Model, etc.
   
2. Get few candidates with similar features
   
3. Get the best candidate.
   

I am having problem with the first step (1). In order to get 'Panasonic => Brand', DMC-FX07EB => Model, silver => color, I need to have index where each token of the product description correspond to certain attribute name (Brand, model, color, etc.) in the existing database. The problem is that in my database product descriptions are presented as one atomic attribute e.g. 'description' (no separated product attributes).

Basically I don't have training data, so I am trying to build index of all product attributes so I can build training data. So far I have attributes from bestbuy.com and semantics3.com APIs, but both sources lack most of attributes or contain irrelevant ones. Any suggestions for better APIs to get product attributes? Better approach to do this?

P.S. For every product there is a matched product description in the Database, which is as well in a form of one atomic attribute. I have checked this question on SO, it helped me and it seems we have same approach but I am still trying to get training data.

Answer 1

Left you a quick response on SO. The gist is that you can collect a lot of information from electronics shops and manufacturers' web sites, and lots you can annotate manually. If your goal is to only get training data, that's all you need:

My answer form the cross-post: "Having developed a commercial analyzer of this kind, I can tell you that there is no easy solution for this problem. But there are multiple shortcuts, especially if your domain is limited to cameras/electronics.

Firstly, you should look at more sites. Many have product brand annotated in the page (proper html annotations, bold font, all caps in the beginning of the name). Some sites have entire pages with brand selectors for search purposes. This way you can create a pretty good starter dictionary of brand names. Same with product line names and even with models. Alphanumeric models can be extracted in bulk by regular expressions and filtered pretty quickly.

There are plenty of other tricks, but I'll try to be brief. Just a piece of advice here: there is always a trade-off between manual work and algorithms. Always keep in mind that both approaches can be mixed and both have return-on-invested-time curves, which people tend to forget. If your goal is not to create an automatic algorithm to extract product brands and models, this problem should have limited time budget in your plan. You can realistically create a dictionary of 1000 brands in a day, and for decent performance on known data source of electronic goods (we are not talking Amazon here or are we?) a dictionary of 4000 brands may be all you need for your work. So do the math before you invest weeks into the latest neural network named entity recognizer."