I would like to know how the alpha parameter, in Multinomial Bayes, affects the text classification task.

I know that this parameter is correlated to the algorithm's ability in classifying unseen words during training. How changes text classification using low or high values of alpha?

  • 2
    $\begingroup$ Are you referring to the alpha parameter in the laplace smoothing ? $\endgroup$ – moksha Apr 18 '18 at 11:16
  • $\begingroup$ @moksha Yes, exactly. $\endgroup$ – Simone Apr 18 '18 at 11:25

Lets assume you are building a text classifier with a training set of 5 sentences. For this example, lets say you are trying to classify tweets (which are usually a sentence long) to whether it was a tweet made by Trump or not. You are given a tweet, "I have huge respect for women" and your goal is to classify it to 'Trump' or 'Not Trump'. Moreover, you are given 5 other random tweets and you also know whether those tweets were by Trump or not (Basically, they are already classified) this is your training set.

in other words you are calculating,

$$P(I\,have\,huge\,respect\,for\,women) = P(I)\times P(have)\times P(huge)\times P(respect)\times P(for)\times P(women)\\$$ $$\\$$ $$P(I\,have\,huge\,respect\,for\,women|Trump) = P(I|Trump)\times P(have|Trump)\times P(huge|Trump)\times P(respect|Trump)\times P(for|Trump)\times P(women|Trump)\\$$ $$\\$$ $$P(I\,have\,huge\,respect\,for\,women|Not\,Trump) = P(I|Not\,Trump)\times P(have|Not\,Trump)\times P(huge|Not\,Trump)\times P(respect|Not\,Trump)\times P(for|Not\,Trump)\times P(women|Not\,Trump)$$ $$\\$$ Depending on the probablity of equations (2) and (3) the user would make a decision whether the statement was made by Trump or not. Now lets say that none of the 5 training set tweets have the word huge in it, in which case, $$\\$$ $$P(huge|Trump)=P(huge|Not\,Trump)=0$$ $$\\$$ And hence, equations (2) and (3) are now zero and this is bad. A solution to this is to do smoothing or rather Laplace smoothing. The basic idea is to increase the probablities of all bigrams in your non-maximum likelihood equation (Since we are changing the counts from what they occurred in hopes to make it better) by 1 to make everything non-zero. That increase by 1 is called adding the pseudocount or as you know it as $\alpha$. Now, $\alpha=1$ may not always give the most accurate probablities and hence they could attain any finite non-negative integer. The way to know what $\alpha$ gives the most accurate responses is through iterating over all values of $\alpha$ on the training set unfortunately. The way to do that is whole different topic.

PS : Trump did say he has huge respect for women. Such irony.

  • $\begingroup$ Change α's value bring any real difference in how the classifier treat sentences with words never seen before? $\endgroup$ – Simone Apr 18 '18 at 17:00
  • 1
    $\begingroup$ @Simone for more complex sentences and by that I mean for sentences that may be more and more out of context relative to the training set, the noise will increase. In such cases increasing α will be needed to smooth out the class separation. We can use cross-validation to find the optimal k because there is an inherent trade-off between the complexity of what we want to fit and the goodness of the fit. $\endgroup$ – moksha Apr 19 '18 at 10:42
  • $\begingroup$ Ok, but how change the behavior of the classifier (if it changes) when we change the value of α? $\endgroup$ – Simone Apr 20 '18 at 17:33
  • $\begingroup$ To test out results for different hyper parameters alpha what values we should be considering? Like for k in KNN we can take values like [3, 15, 25, 51, 101] $\endgroup$ – Dipen Gajjar Dec 21 '19 at 5:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.