# Best way to extract information from text description and match it with set of words

I have 10k records of data, each record represents a unique product(10k class labels) and its description. For example, "Coffee Maker, this product takes coffee beans and brew it, to make tasty cofe". Description can be little bigger and smaller. Most of the data also mentions acronyms in the description to refer the product or some other related product and yeah there spelling mistakes too.

So what is the best approach to solve this problem, im open for using Machine Learning/Deep Learning. Please help me on how to build a model, that takes a small description like three or four words or more as a input and suggest a list of products that closely represent those words.

• How many unique labels do you have? – JahKnows May 15 '18 at 14:39
• 10 thousand records of data and each record is a unique label. – Sivabushan May 15 '18 at 14:40
• Then this is not a machine learning/deep learning problem. What do you want to do in the future with your trained model? Do you want to find the most similar instance. – JahKnows May 15 '18 at 14:43
• i need to it for searching and bringing up the most relevant products for a search string. suppose i search for "Coffee Maker", i should able to see all the products related to that string. Challenge is "coffee maker" may not be explicitly mentioned in our description. So we need to build a smart agent that can understand the input text and get the best possible products for that inventory – Sivabushan May 15 '18 at 14:49
• This is not a machine learning problem in the least. Machine learning is meant to fit distributions. You cannot learn a probability distribution from a single instance. But there are some more traditional means by which to do this. I do strongly suggest reading some basics of machine learning before attempting to apply it blindly especially in cases where it does not apply. – JahKnows May 15 '18 at 15:30

You can solve this problem using traditional recommendation system algorithms for text. I will show you two ways how this can be implemented, one using a simple linear search and the other using some clustering approach. First, you will need to vectorize your descriptions.

# Feature extraction

You will build a dictionary of existing words or character sequences and then you will fill this vector with the number of occurances in your short description. There are two techniques, I would recommend for this, the first is bag-of-words, don't forget to only use the stem of words since coffee should be the same as coffees. The second is n_grams which is sequences of characters.

## n-grams

n-grams is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...

The segmentation works as follows:

The String: Hello World

2-gram: "He", "el", "ll", "lo", "o ", " W", "Wo", "or", "rl", "ld" 3-gram: "Hel", "ell", "llo", "lo ", "o W", " Wo", "Wor", "orl", "rld" 4-gram: "Hell", "ello", "llo ", "lo W", "o Wo", " Wor", "Worl", "orld"

Thus in your example, if we use 4-grams, truncations of the word Hello would appear to be the same. And this similarity would be captured by your features.

## Bag-of-words

This builds a dictionary of the words it has seen during the training phase. Then using the word the frequency of each word in the example a vector is created. This can then be used with any standard machine learning technique.

For example: If the dictionary includes the words {car, coffee, motel, hotel, world, van, soup} and we have the description "the soup in this motel is amazing!". Then the resulting vector would be $[0, 0, 1, 0, 0, 0, 1]$.

You will now have a vector representation of each of your descriptions. Then for a search term like "yummy food motel", you can vectorize it as $[0, 0, 1, 0, 0, 0, 0]$, then find the most similar instance in your set to this vector. You can use the Euclidean distance as a metric of similarity. This method can be considered a k-nearest neighbors (k-NN) approach.