-2
$\begingroup$

I have 10k records of data, each record represents a unique product(10k class labels) and its description. For example, "Coffee Maker, this product takes coffee beans and brew it, to make tasty cofe". Description can be little bigger and smaller. Most of the data also mentions acronyms in the description to refer the product or some other related product and yeah there spelling mistakes too.

So what is the best approach to solve this problem, im open for using Machine Learning/Deep Learning. Please help me on how to build a model, that takes a small description like three or four words or more as a input and suggest a list of products that closely represent those words.

$\endgroup$
  • $\begingroup$ How many unique labels do you have? $\endgroup$ – JahKnows May 15 '18 at 14:39
  • $\begingroup$ 10 thousand records of data and each record is a unique label. $\endgroup$ – Sivabushan May 15 '18 at 14:40
  • $\begingroup$ Then this is not a machine learning/deep learning problem. What do you want to do in the future with your trained model? Do you want to find the most similar instance. $\endgroup$ – JahKnows May 15 '18 at 14:43
  • $\begingroup$ i need to it for searching and bringing up the most relevant products for a search string. suppose i search for "Coffee Maker", i should able to see all the products related to that string. Challenge is "coffee maker" may not be explicitly mentioned in our description. So we need to build a smart agent that can understand the input text and get the best possible products for that inventory $\endgroup$ – Sivabushan May 15 '18 at 14:49
  • $\begingroup$ This is not a machine learning problem in the least. Machine learning is meant to fit distributions. You cannot learn a probability distribution from a single instance. But there are some more traditional means by which to do this. I do strongly suggest reading some basics of machine learning before attempting to apply it blindly especially in cases where it does not apply. $\endgroup$ – JahKnows May 15 '18 at 15:30
0
$\begingroup$

You can solve this problem using traditional recommendation system algorithms for text. I will show you two ways how this can be implemented, one using a simple linear search and the other using some clustering approach. First, you will need to vectorize your descriptions.

Feature extraction

You will build a dictionary of existing words or character sequences and then you will fill this vector with the number of occurances in your short description. There are two techniques, I would recommend for this, the first is bag-of-words, don't forget to only use the stem of words since coffee should be the same as coffees. The second is n_grams which is sequences of characters.

n-grams

n-grams is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...

The segmentation works as follows:

The String: Hello World

2-gram: "He", "el", "ll", "lo", "o ", " W", "Wo", "or", "rl", "ld" 3-gram: "Hel", "ell", "llo", "lo ", "o W", " Wo", "Wor", "orl", "rld" 4-gram: "Hell", "ello", "llo ", "lo W", "o Wo", " Wor", "Worl", "orld"

Thus in your example, if we use 4-grams, truncations of the word Hello would appear to be the same. And this similarity would be captured by your features.

Bag-of-words

This builds a dictionary of the words it has seen during the training phase. Then using the word the frequency of each word in the example a vector is created. This can then be used with any standard machine learning technique.


For example: If the dictionary includes the words {car, coffee, motel, hotel, world, van, soup} and we have the description "the soup in this motel is amazing!". Then the resulting vector would be $[0, 0, 1, 0, 0, 0, 1]$.


Applying your search

You will now have a vector representation of each of your descriptions. Then for a search term like "yummy food motel", you can vectorize it as $[0, 0, 1, 0, 0, 0, 0]$, then find the most similar instance in your set to this vector. You can use the Euclidean distance as a metric of similarity. This method can be considered a k-nearest neighbors (k-NN) approach.

$\endgroup$
  • $\begingroup$ how do we tackle similar data situtations like coffee machine - caffine maker- cappucino brewer. My example seems little lame, but problems something like that. $\endgroup$ – Sivabushan May 15 '18 at 16:57
  • $\begingroup$ Consider how a human who has never heard the english language would do this. Obviously a computer would not be able to make an equivalency between cappucino and coffee as it currently stands. There are some ways by which this can be done using very few words and a very large dataset. However for the scope of your project none of the existing algorithms are feasible. Unless you build a tailored dictionary for all word equivalencies. $\endgroup$ – JahKnows May 15 '18 at 17:10
  • 1
    $\begingroup$ I came to this conclusion yesterday, My bad, i didnot explain my thought process in the question. i didn't want someone to pick up my solution and help me refine it, i wanted to see new approaches people would suggest. Thank you for taking time to work on my problem. $\endgroup$ – Sivabushan May 15 '18 at 17:27
  • $\begingroup$ @Sivabushan, next time please indicate your past ideas in your post so we avoid typing up what you have already considered. $\endgroup$ – JahKnows May 15 '18 at 17:29
  • $\begingroup$ Sure buddy, im currently thinking of two things. Can we use word2vec to solve this problem and also topic modelling. do you have anything to say about that? $\endgroup$ – Sivabushan May 15 '18 at 17:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.