5
$\begingroup$

I'm training a character based RNN model for text prediction and want to compare it to similar models. Since most literature uses word based perplexity as a performance metric, what would be the "proper" way to calculate word based perplexity from a character based model?

$\endgroup$
2
$\begingroup$

Actually, there is a formula which can easily convert character based PPL and word based PPL.

$PPL = 2^{(BPC*Nc/Nw)}$

where $BPC$ is character based $PPL$, $Nc$ and $Nw$ are the number of characters and words in a test set, respectively.

The formula is not completely fair, but it at least offers a way to comparing them. The following are some reference.

[1] Hwang K, Sung W. Character-Level Language Modeling with Hierarchical Recurrent Neural Networks[J]. 2016.

[2] Graves A. Generating Sequences With Recurrent Neural Networks[J]. Computer Science, 2013.

[3] T. Mikolov, I. Sutskever, A. Deoras, H. Le, S. Kombrink, and J. Cernocky.Subword language modeling with neural networks. Technical report, Un-published Manuscript, 2012.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.