Tokenize sentence based on a dictionary

Question

I have few key words say - RAM, Speaker, Brand, Display etc. I have made a dictionary of all possible values these keys can have. Like - RAM can have 12, 12gb, 12GB; Display can have 12inch, 12", 12 inch.

I am given a title say - "Apple iPhone 5s (Space Grey, 16GB)"

I have to extract tokens from these titles and map to the dictionary values and assign appropriate keyword to each token.

Like for this - Brand - Apple, Color - Grey, Storage - 16GB

How should i tokenize these sentences? Doing it just by space will not be enough- like in this title - "Samsung Guru Music 2 SM-B310E (White)" 2 will be one token and can map to RAM,Display anything.

Will any NLP library help with this? I am using python to code and new to NLP.

Answer 1

Look at the NLTK library for Python, there are functions to facilitate tokenizing sentences.

Answer 2

If a couple of words appears together many times in your corpus, e.g., 'new', 'york', you can use this. It works with more than 2 tokens. This way you have one token called "new_york"

Also for your dict, you can use this, it does exactly what you are looking for.

Answer 3

This seems like it might a good fit to regular expressions but lets first talk about how to restrict the search space so you don't have to use regex. Before you even start though you need to make sure your requirements are possible - it looks like you want to match 16 to storage and 12 to screen size without requiring GB or inches at the end, which means you need to find some way to distinguish those numbers. Here's a few ideas of how to make these restrictions:

Storage: this is very likely to be powers of 2, (8, 16, 32, 64...) so you only really need to check if the string has a power of two in it:

[storage in "iPhone 5s 16 GB" for storage in ["8", "16", "32", "64"]]

will return you [False, True, False, False], telling you that there's a 16 in there.

Display: again, these are likely to be quite restricted - as you say, 12, 13, etc. You could probably get away with something similar to my storage trick above and ignore trying to find inches.

Brand, Model: These should be really simple - again, make a list of the brands you're searching for, and then do the list comprehension above.

If this doesn't work, you can try regexes. These are more complex and usually more trouble than they're worth - they're hard to read, maintain and I've often got bitten by them. However:

The regex (?i)[0-9]+(\s|)GB in python will do a case insensitive ((?i)) search for strings containing one of more numbers 0-9 ([0-9]+), zero or more white space (\s*) and then the letters GB (GB). GB will be matched in upper or lower case because of the (?i) option are the start. Calling

re.search('(?i)[0-9]+\s*GB', "Apple iPhone 5s (Space Grey, 16GB)")

will return a match object giving you the matched string.

I find regexs quite complicated so general use a helpful website to at least try catch bugs before I code them.

Answer 4

I've been working on a very similar problem extracting materials details from invoicing information. One approach that has shown promise has been creating a dictionary of lists of dictionaries for each "staple" token - in your case I believe this would be the dictionary of all possible values. Each token is a key with each value being a list of dictionaries, with each of these dictionaries being comprised of all other tokens found in the sentence of the top level key, their frequency seen together, and the (average) distance from the top level key in each sentence. With this, I was able to generate a probability distribution depicting the likelihood of each sub level token's presence and placement in the sentence relative to the top level token/key.

This has shown promise with identifying what measure the numeric tokens represent in a sentence when present with a top level token/key. In your example above, it may even result in a low level confidence that the "2" token actually maps to anything, which you can code in logic to flag. For instance, I discovered that if a material could have dimensions length x height x depth, it was most likely that any given numeric measure would be length or height, and that only a very small amount of the time it could be depth. I then expanded that logic to "step forward" and use the sub level's dictionary (ie length) to further extrapolate the likelihood of the next numeric token identified, and so on.

This also led to being able to generalize characteristics in sentences based on the source/publisher (in your case this may correlate to the manufacturer's preferred method of naming convention.)

If you've got to use reflex, pythex.org is also an extremely useful tool for testing. Just remember that pythex operates under the r'' assumption, so escaping special characters may have different results in your compiler.

This approach also assumes you have a large amount of data with which to experiment -if your corpus is too small or the same words are repeated too often, many NLP/ngram/common words approaches aren't going to very well.