10
$\begingroup$

I am looking to design a system that given a paragraph of text will be able to categorize it and identify the context:

  1. Is trained with user generated text paragraphs (like comments/questions/answers)
  2. Each item in the training set will be tagged with . So for e.g. ("category 1", , "text paragraph")
  3. There will be hundreds of categories

What would be the best approach to build such a system? I have been looking at a few different options and the following is a list of possible solutions. Is Word2Vec/NN the best solution at the moment?

  1. Recursive Neural Tensor Network fed with averaged Word2Vec data
  2. RNTN and The Paragraph Vector (https://cs.stanford.edu/~quocle/paragraph_vector.pdf)?
  3. TF-IDF used in a Deep Belief Network
  4. TF-IDF and Logistic Regression
  5. Bag of words and Naive Bayes classification
Improve this question
$\endgroup$
3
  • $\begingroup$ Can you clarify what kind of categories? Will it need to be able to handle new categories and/or unseen words? The requirements regarding infrequent terms and unseen categories will help the design of the system. $\endgroup$
    – rabbit
    Commented Nov 4, 2015 at 19:27
  • $\begingroup$ Thanks @NBartley. Unseen words will also be a high probability. The input paras will be user generated content, hence the possibility of new unseen words will be very high. The categories would be defined, but we will need to expand the category list over time. Thanks $\endgroup$
    – Shankar
    Commented Nov 5, 2015 at 5:57
  • $\begingroup$ You should check out sense2vec too arxiv.org/abs/1511.06388. In a nutshell it's word embeddings combined with Part-Of-Speech tagging. It's reported it made word embeddings more accurate by disambiguating homonyms. It would be interesting to see if it also improves performance in classification tasks. $\endgroup$
    – wacax
    Commented Dec 5, 2015 at 20:50

1 Answer 1

Reset to default
5
$\begingroup$

1) Max-Entropy(Logistic Regression) on TFIDF vectors is a good starting point for many NLP classification task.

2) Word2vec is definitely something worth trying and comparing to model 1. I would suggest using the Doc2Vec flavor for looking at sentences/paragraphs.

Quoc Le and Tomas Mikolov. Distributed Representations of Sentences and Documents. http://arxiv.org/pdf/1405.4053v2.pdf

Gensim(python) has a nice Doc2vec model.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks @rushimg. If the categories are closely related, i.e. the para of text that are used as input have a large amount of common words, which of the two approaches would be better at understanding the context and differentiating between the two? $\endgroup$
    – Shankar
    Commented Nov 5, 2015 at 6:02
  • $\begingroup$ I would use the Doc2Vec model due to the fact that it removes the bag-of-words assumption of the max-ent model. If tf-idf is used as features in the max-ent model then this would also reduce the impact of common words. I think trying out both methods and tweaking them would be the best course of action. $\endgroup$
    – rushimg
    Commented Nov 5, 2015 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.