What are key dataset requirements for topic models and word embeddings?

Question

I have a dataset of 2000 documents where avg doc size is 300 words. The vocab is dominated by domain-specific words.

My goal is to find similar documents. For this, I tried LDA, LSI, Doc2Vec (topics=100) but results are not great. LSI is better than the others in my dataset. I also tried word2vec (size=100) & word movers distance but again no luck. I am thinking of trying POS tags & then building some ontology model but not sure.

Are my results poor because of a bad dataset? What are some other techniques that I can try?

Answer 1

Similarity of documents can be done with varied approaches.

As your documents are based on domain based words, you could employ a tfidf representation for each document and compute similarity based on this.

Previously, I have used word2vec representations of words and constructed document vectors by taking an average of all the vectors ( as one of the approaches ) and did a cosine similarity between these vectors. Basically an n*n similarity computation.

When you do topic modelling, even though the idea is to find documents which have similar topic distribution, there is a prior to the approach i.e identifying number of topics. Just using randomly 100 topics without looking at the distribution of the words might lead you on a wrong path.

A very interesting approach I have recently come across is combining topic modelling and word vectors, here is the link to a blog by stitchfix : http://multithreaded.stitchfix.com/blog/2016/05/27/lda2vec/

Answer 2

If you are unsure of determining number of topics which has to be passes as a piori. You could use coherence techniques either C_V or u_mass to get optimal number of topics for the data set.

Answer 3

One reason for the poor word embeddings could be the small number of documents. I think it's worth trying pre-trained word vectors (e.g. google news vectors: https://github.com/mmihaltz/word2vec-GoogleNews-vectors). You do mention a large number of domain-specific words, which could be an issue, but these vectors are trained with a very large vocabulary.

Otherwise I would second the idea of using something simpler like tf-idf scaled word frequencies, maybe limiting the vocabulary and reducing dimensionality.