I'm training a character based RNN model for text prediction and want to compare it to similar models. Since most literature uses word based perplexity as a performance metric, what would be the "proper" way to calculate word based perplexity from a character based model?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Actually, there is a formula which can easily convert character based PPL and word based PPL.
$PPL = 2^{(BPC*Nc/Nw)}$
where $BPC$ is character based $PPL$, $Nc$ and $Nw$ are the number of characters and words in a test set, respectively.
The formula is not completely fair, but it at least offers a way to comparing them. The following are some reference.
[1] Hwang K, Sung W. Character-Level Language Modeling with Hierarchical Recurrent Neural Networks[J]. 2016.
[2] Graves A. Generating Sequences With Recurrent Neural Networks[J]. Computer Science, 2013.
[3] T. Mikolov, I. Sutskever, A. Deoras, H. Le, S. Kombrink, and J. Cernocky.Subword language modeling with neural networks. Technical report, Un-published Manuscript, 2012.
