I have a big collection of phrase segments (not whole ones) with user provided labels based on text similarity:
train_data (labeled phrase segments):
"has watched movie" => phraselabel1:"movie watch activity"
"watched with friends X,Y,Z movie" => phraselabel1:"movie watch activity"
"watched show" => phraselabel2: "show watch activity"
"is jobless" => phraselabel3: "lack of job"
input phrase segment:
"watched movie in the past" => ??
output:
recommend ordered set of phrase labels: phraselabel1 then phraselabel3 based on similarity criteria
I would like to build a (scalable) experimental recommendation system that finds the most similar phrase segments that already have phrase labels to the input unlabeled one and return the best labels based on the similarity. Labels will be added by users to subset of the data initially and the system will help future users with existing labels.
- The number of phrase-labels will be way less than then total number of phrase segments but more granular than topics/tags
I was thinking of the following approaches:
- Do TF-IDF (http://spark.apache.org/docs/latest/mllib-feature-extraction.html) between input phrase segment and all the labeled phrase segments. By finding the most similar I can derive the labels that are best to be suggested
- Do TF-IDF between input phrase segment and all the phrase-labels.
- Word2Vec?? I am not sure if it can be used for phrase segments as input and not just words..
- Labeled LDA (Latent Dirichlet Allocation (https://github.com/myleott/JGibbLabeledLDA) although I don't think this is readily available in Spark
- Neural Network (https://spark.apache.org/docs/latest/ml-classification-regression.html#multilayer-perceptron-classifier) trained on the words of all input phrase segments as inputs and phraselabels as outputs using the user click feedback on the selected phraselabel for back-propagation training.
I would appreciate if you could shed some light on which approach (from above or a new one) is better based on the use case..
Thanks :)