1
$\begingroup$

I'm using this formula to covert subword perpexity to word perplexity: PPL_word = exp(log(PPL_subword) * num_subwords / num_words) The question is do I need to include the [SEP] and [CLS] tokens when counting subwords?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.