What is the difference between word-based and char-based text generation RNNs?

Question

While reading about text generation with Recurrent Neural Networks I noticed that some examples were implemented to generate text word by word and others character by character without actually stating why.

So, what is the difference between RNN models that predict text per-word basis and the ones that predict text per-char basis? Do word-based RNN require a bigger corpus size? Do char-based RNN generalize better? Maybe the only difference is input representation (one-hot encoding, word embeddings)? Which ones to choose for text generation?

Answer 1

Here is what I learnt recently.

Obviously, when talking about text generation RNNs we are talking about RNN language models. When asking about word/char-based text generation RNNs, we are asking about word/char-based RNN language models (LM).

Word-based LMs display higher accuracy and lower computational cost than char-based LMs.

This drop of performance is unlikely due to the difficulty for character level model to capture longer short term memory, since also the Longer Short Term Memory (LSTM) recurrent networks work better with word-based input.

This is because char-based RNN LMs require much bigger hidden layer to successfully model long-term dependencies which means higher computational costs.

Therefore, we can say that

one of the fundamental differences between the word level and character level models is in the number of parameters the RNN has to access during the training and test. The smaller is the input and output layer of RNN, the larger needs to be the fully connected hidden layer, which makes the training of the model expensive.

However, char-based RNN LMs better model languages with a rich morphology such as Finish, Turkish, Russian etc. Using word-based RNN LMs to model such languages is difficult if possible at all and is not advised.

The above analysis makes sense especially when you look at the output text, generated by char-based RNNs:

The surprised in investors weren’t going to raise money. I’m not the company with the time there are all interesting quickly, don’t have to get off the same programmers.

While simple char-based Maximum Likelihood LM with a 13-character window delivers this:

And when she made many solid bricks. He stacked them in piles and stomped her feet. The doctor diagnosed him with a bat. The girl and her boyfriend asked her out.

Of course I cherry-picked the example (actually most ML LM examples looked better than any RNN generated text I've read so far) and this tiny ML LM was trained on a simpler corpus but you get the idea: straightforward conditional probability generates better texts than far more complex char-based RNN.

Char-based RNN LMs can mimic grammatically correct sequences for a wide range of languages, require bigger hidden layer and computationally more expensive while word-based RNN LMs train faster and generate more coherent texts and yet even these generated texts are far from making actual sense.

Answer 2

There is a nice write up about Language modeling a billion words. Below are some excerpts:

Word-level models have an important advantage over character-level models.
Take the following sequence as an example (a quote from Robert A. Heinlein):

Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something.

After tokenization, the word-level model might view this sequence as containing 22 tokens. On the other hand, the character-level will view this sequence as containing 102 tokens. This longer sequence makes the task of the character model harder than the word model, as it must take into account dependencies between more tokens over more time-steps. Another issue with character language models is that they need to learn spelling in addition to syntax, semantics, etc. In any case, word language models will typically have lower error than character models

The main advantage of character over word language models is that they have a really small vocabulary. For example, the GBW dataset will contain approximately 800 characters compared to 800,000 words (after pruning low-frequency tokens). In practice this means that character models will require less memory and have faster inference than their word counterparts. Another advantage is that they do not require tokenization as a preprocessing step.

Answer 3

In my opinion, the character based RNNs will also perform better but they need much more data than the word based models and character based models need to train for a much longer period of time. I would say it is more of a trial and error as well as a trade-off between data and computation power available.

Answer 4

Same: Both charRNN and wordRNN are auto-regressive models

Diff:

1. CharRNN uses a smaller vocab table than WordRNN, with little OOV (out of vocabulary) occurrence. Whereas wordRNN needs to handle OOV explicitly to avoid zero probability. This also means wordRNN needs to handle vocab table carefully to avoid data leakage. (using words from test data only is essentially cheating.)
2. CharRNN needs more autoregressive steps than wordRNN.