1
$\begingroup$

Do people use n-grams or 1,2,3,...n-grams in both matrix factorisation and generative models in Topic Modeling?

I've been trying to understand the basics of Topic Modeling and came to know that there are two ways - Matrix Factorisation like LSA and NNMF and generative models like LDA and pLSA.

However, while reading the texts, I had a question - Do people use n-grams or 1,2,3,...n-grams in both matrix factorisation and generative models in Topic Modeling? For example, if n=5, then do people use only 5-grams or do they use all unigrams, bigrams, trigrams, 4-grams and 5-grams for creating the document term matrix?

If there are contextual answers then what are the reasons for using either?

Thanks in advance.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

It is best to use 1,2,3,...n-grams. Giving the model more features allows it to better learn patterns in the data. Often a threshold for the number of occurrences is used to filter out infrequent ngrams.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for sharing this info. Using the combination of different levels of n-grams definitely sounds tempting. But what are the downsides of this approach? Does this affect the conditional probabilities? Does this impact the topic distribution? $\endgroup$
    – rahuladwani
    Dec 5, 2022 at 6:03
0
$\begingroup$

In the case of traditional topic modelling approaches like LSA, LDA, etc., no, it's done only with unigrams. These methods rely on word cooccurrences in order to cluster semantically meaningful topics. In a regular text most words appear only once or twice and therefore are not usable as cooccurrences. If one considers n-grams instead of unigrams, there would be very few cooccurrences left in a text, so it's very unlikely that these models would be able to correctly cluster these n-grams by topic.

Mixing different levels of n-grams might work (not sure if it's been done before?), but I suspect that this could cause inconsistencies in the conditional probabilities word given topic.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for answering my question. I have a quick follow-up: what kind of inconsistencies are likely to be caused if a combination of different levels of n-grams is used? If possible, please add an example. $\endgroup$
    – rahuladwani
    Dec 5, 2022 at 6:07
  • $\begingroup$ @rahuladwani In the regular case of unigrams, the probs $p(w|t)$ are comparable across words and naturally sum to 1 (since one can divide by the total number of words for $p(w)$). I don't know how this is handled with a combination of n-grams, since there's no natural universe for the probability space. But apparently from Brian's answer it seems to be common, so I assume that this issue is solved somehow. $\endgroup$
    – Erwan
    Dec 5, 2022 at 11:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.