-1
$\begingroup$

In "August 2019 draft of the 3rd edition of Jurafsky & Martin Speech and Language Processing" book's section 3.5 (Kneser-Ney Smoothing) it is stated that

The astute reader may have noticed that except for the held-out counts for 0 and 1, all the other bigram counts in the held-out set could be estimated pretty well by just subtracting 0.75 from the count in the training set! Absolute discounting formalizes this intuition by subtracting a fixed (absolute) discount d from each count

enter image description here The part I don't understand is that intuitively, if the two corpuses are from the same topic, then the distribution of n-gram counts had to be at least similar to each other, and if they are about different topics, the distribution had to be much different. How it comes that the distribution of counts of n-grams differs by a fixed number, and how are we guaranteed that it will always hold?

|improve this question
$\endgroup$
  • $\begingroup$ How is the held-out set defined? And why are the count in the held-out set not integers, is it an average across several random splits? $\endgroup$ – Erwan Nov 9 at 1:50
  • $\begingroup$ @Erwan Held-out set is defined as follows: A held-out corpus is an additional training corpus that we use to set hyperparameters like these λ values, by choosing the λ values that maximize the likelihood of the held-out corpus. $\endgroup$ – Grigor Bezirganyan Nov 9 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.