I'm using this formula to covert subword perpexity to word perplexity:
PPL_word = exp(log(PPL_subword) * num_subwords / num_words)
The question is do I need to include the [SEP] and [CLS] tokens when counting subwords?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community