I want to fit the parameters of my language model by minimizing the entropy/ maximizing likelihood of my language model on my data-set. However, I am uncertain as how I should go about doing this. Should I calculate the entropy of each sentence, then look at the average entropy over all sentences? Or, should I instead look at minimizing the entropy of the entire text file. Is there even a difference?
Personally I feel like the correct answer is to consider the entropy of the entire file. However I was told that I should "treat each line as a separate sequence for modelling purposes (rather than having the entire file as a single sequence)". I just don't understand the logic behind that suggestion. Any help would be appreciated.