I want to fit the parameters of my language model by minimizing the entropy/ maximizing likelihood of my language model on my data-set. However, I am uncertain as how I should go about doing this. Should I calculate the entropy of each sentence, then look at the average entropy over all sentences? Or, should I instead look at minimizing the entropy of the entire text file. Is there even a difference?

Personally I feel like the correct answer is to consider the entropy of the entire file. However I was told that I should "treat each line as a separate sequence for modelling purposes (rather than having the entire file as a single sequence)". I just don't understand the logic behind that suggestion. Any help would be appreciated.

Your Answer


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.