1
$\begingroup$

I have a big collection of phrase segments (not whole ones) with user provided labels based on text similarity:

train_data (labeled phrase segments): 
"has watched movie" => phraselabel1:"movie watch activity"
"watched with friends X,Y,Z movie" => phraselabel1:"movie watch activity"
"watched show" => phraselabel2: "show watch activity"
"is jobless" => phraselabel3: "lack of job"

input phrase segment: 
"watched movie in the past" => ?? 

output:
recommend ordered set of phrase labels: phraselabel1 then phraselabel3 based on similarity criteria

I would like to build a (scalable) experimental recommendation system that finds the most similar phrase segments that already have phrase labels to the input unlabeled one and return the best labels based on the similarity. Labels will be added by users to subset of the data initially and the system will help future users with existing labels.

  • The number of phrase-labels will be way less than then total number of phrase segments but more granular than topics/tags

I was thinking of the following approaches:

I would appreciate if you could shed some light on which approach (from above or a new one) is better based on the use case..

Thanks :)

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

CNN-Text-classification. Try this. Keep in mind that a deep learning algorithm will require a large data-set if you want a good accuracy. Since you want to give ranked recommendations to the users, a soft-max layer at the end will give you the probability of the phrase belonging to all the classes, so you can rank the labels in the order of the probability and suggest them to the user.

Your first approach is the simplest and best given that you have finite data. I don't think it will be very scalable as your data grows. Also this approach will work best only when your phrases are so limited in their context and vocabulary. Using a convnet you can train on bigger and more ambiguous phrases and get good predictions.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks! I will have a detailed look shortly! So you are suggesting that Spark won't be enough to train any of the other models offline and then be used online with low latency? $\endgroup$
    – Michail Michailidis
    Dec 20, 2016 at 18:20
  • 1
    $\begingroup$ It might be. No way to know unless you run a model and look at the performance metrics. But again, its the era of deep learning, when machines can learn contexts why just teach them words then. $\endgroup$
    – Himanshu Rai
    Dec 20, 2016 at 18:58
  • $\begingroup$ I am a bit concerned about black boxes like neural networks /deep learning etc.. they work but nobody will ever know why :P $\endgroup$
    – Michail Michailidis
    Dec 20, 2016 at 19:02
  • 1
    $\begingroup$ Can't say about never. But again interpretability is not the goal. In all simplicity can you tell how do you learn to differentiate a cat from a dog? The goal is to teach a machine to learn from mistakes without us telling it what features to look for. That is intelligence. $\endgroup$
    – Himanshu Rai
    Dec 21, 2016 at 4:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.