How can Dirichlet smoothing be used as an IDF component to estimate the probabilities of a Topic model ? i.e, Smoothing with a background collection model to estimate topic model ?

I've seen many articles suggesting that Smoothing serves two purposes :

  1. To avoid the zero-probability scenario for words

  2. To act as a discriminative component (IDF) along with a collection model to avoid giving higher probabilities to common words like , 'the' , 'a' etc while estimating a topic model

What I don't understand is , how can Dirichlet smoothing , as shown below ( Estimation of word in a topic model )

enter image description here achieve the effect of IDF and ensure common words like the , a etc receive lower probabilities in a topic model ?

I've worked out some examples but I don't get lesser probabilities for such common words . In fact , we are only adding pseudo counts to every word. So , how is IDF done using Dirichlet ?

I'm finding it hard to understand this . Please explain in detail .

Thanks in advance !


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.