9
$\begingroup$

I want to use Latent Dirichlet Allocation for a project and I am using Python with the gensim library. After finding the topics I would like to cluster the documents using an algorithm such as k-means(Ideally I would like to use a good one for overlapping clusters so any recommendation is welcomed). I managed to get the topics but they are in the form of:

0.041*Minister + 0.041*Key + 0.041*moments + 0.041*controversial + 0.041*Prime

In order to apply a clustering algorithm, and correct me if I'm wrong, I believe I should find a way to represent each word as a number using either tfidf or word2vec.

Do you have any ideas of how I could "strip" the textual information from e.g. a list, in order to do so and then place them back in order to make the appropriate multiplication?

For instance the way I see it if the word Minister has a tfidf weight of 0.042 and so on for any other word within the same topic I should be to compute something like:

0.041*0.42 + ... + 0.041*tfidf(Prime) and get a result that will be later on used in order to cluster the results.

Thank you for your time.

$\endgroup$
  • 1
    $\begingroup$ As explained in the tutorial, you can express documents as vectors. Cluster those vectors. $\endgroup$ – Emre Nov 13 '14 at 18:54
  • $\begingroup$ I know mate but I have to cluster them according to the topics created after I apply LDA on my collection. Each topic should be represented as a vector in order to compare each document with each topic and find the correspondent topic or topics for each doc. $\endgroup$ – Swan87 Nov 14 '14 at 11:03
  • $\begingroup$ You don't have to represent each word as a vector. You get the new representation for the entire document by applying the LDA transformation you learned to the corpus. For an example with LSI, see this link: radimrehurek.com/gensim/tut2.html The key part is where they apply the learned LSI transformation to the entire corpus with lsi[doc_bow] $\endgroup$ – Will Stanton Jun 13 '15 at 2:29
4
$\begingroup$

Assuming that LDA produced a list of topics and put a score against each topic for each document, you could represent the document and it's scores as a vector:

Document | Prime | Minister | Controversial | TopicN | ...
   1       0.041    0.042      0.041          ...
   2       0.052    0.011      0.042          ...

To get the scores for each document, you can run the document. as a bag of words, through a trained LDA model. From the gensim documentation:

>>> lda = LdaModel(corpus, num_topics=100)  # train model
>>> print(lda[doc_bow]) # get topic probability distribution for a document

Then, you could run the k-means on this matrix and it should group documents that are similar together. K-means by default is a hard clustering algorithm implying that it classifies each document into one cluster. You could use soft clustering mechanisms that will give you a probability score that a document fits within a cluster - this is called fuzzy k-means. https://gist.github.com/mblondel/1451300 is a Python gist showing how you can do it with scikit learn.

ps: I cant post more than 2 links

$\endgroup$
  • $\begingroup$ I tried to do that for "n" no of documents say where topics are t. However for say x no of documents , all t topics prob do not shows up just some ( t- no) topics prob shows up, where 1<=no< t. It does not happen when I run the experiment on small document size. Is it because it doesn't print at all if the prob is 0 ? $\endgroup$ – Manish Ranjan Mar 7 '16 at 17:40
0
$\begingroup$

Complementary to the previous answer you should better not just run kmeans directly on the compositional data derived from the lda topic-doc distribution, instead use some compositional data transformation to project them to the euclidean space like ilr or clr.

(Example)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.