1
$\begingroup$

I have a description of a product say mobile in this form -

"5.5" (13.97cm) full hd 1920 x 1080 pixels ips screen gorilla glass display with 401ppi resolution - 450 nits brightness & contrast ratio of 1000:1first phone with theatermax technology and dolby atmos surround sound over speakers13mp f2.2 primary camera with 5p lens element, dual led dual tone flash, 5mp front camera with 4p lens elementandroid v5.1 os with 1.3 ghz mediatek 6753 64-bit octa core processor, arm mali t720 gpu, 3 gb ddr3 ram, 16gb internal memory (expandable up to 128gb) and dual micro sim dual standby (4g+4g)3300mah lithium-ion battery with fast charger 2.0 providing talk-time of 29 hours"

I have to extract the important elements of the description and map it to keys like -

  • Display size - 5.5
  • Display Resolution - 1920 x 1080
  • Sim type - Dual Sim
  • Ram - 3gb
  • Storage - 16gb.

I have a dictionary that maps key values pairs.

Example - {Ram:["3gb","3 GB","4 gb"],"Sim type":["Dual","Micro","Nano"]}

Can anyone suggest how to do this. I am coding in python. How to proceed it with NLTK. Should I use ngrams??? Any useful examples that uses self tagging will be helpful. All examples i see uses already defined tags like noun,person,organisation etc. I want to give my own dictionary as corpus and train the model so that it extracts the relevant key values pairs from the text.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

You have two options:

  1. Use regular expressions to extract features of interest based on your dictionary. You may have mixed results, since, for example, dual sim cards will be micro or nano, so you will get two types from single description. However, that just says, that the dictionary isn't well structured.
  2. Train your own NER (named entity recognition) system. For that, first, you will need a data set which will consist of your descriptions and marked entities. This would require manual labour, either yours or paid one one some crowd sourcing platform. When you will have a marked-up dataset, you could use NLTK to train a NER. Markup may be done in any way familiar to you, but, generally speaking, xml-like markup is the best choice since there are many ways to train a NER, and most of frameworks know how to parse xml-like tags:

Example markup:

<screen_size>5.5"</screen_size> (13.97cm) full hd <resolution>1920 x 1080</resolution> pixels ips screen gorilla glass display etc.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks a lot. I was thinking on the same line, but wanted to know the best way possible. We have parsed title of the description using Regex, but for full description it breaks a lot because of different type of description patterns. NER requires manual work and since we are a very small team, its expensive. Can I use a ngram Lookup from the dictionary? $\endgroup$
    – maggs
    Mar 31, 2016 at 3:15
  • $\begingroup$ Well, regular expressions allow you to do any lookup, literally/ You can mix all the lookup types and mix them in any way you'd want. I would try to manually markup at least 100 documents and fine-tune your regexps test-driven way to increase precision and recall of matching. $\endgroup$
    – chewpakabra
    Mar 31, 2016 at 8:51
1
$\begingroup$

I think using regexp after some pre-processing (removing stopwords, symbols or lowercasing, etc) is a good solution here. Training a custom NER is too much work.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.