19
$\begingroup$

While reading about text generation with Recurrent Neural Networks I noticed that some examples were implemented to generate text word by word and others character by character without actually stating why.

So, what is the difference between RNN models that predict text per-word basis and the ones that predict text per-char basis? Do word-based RNN require a bigger corpus size? Do char-based RNN generalize better? Maybe the only difference is input representation (one-hot encoding, word embeddings)? Which ones to choose for text generation?

$\endgroup$
1

4 Answers 4

19
$\begingroup$

Here is what I learnt recently.

Obviously, when talking about text generation RNNs we are talking about RNN language models. When asking about word/char-based text generation RNNs, we are asking about word/char-based RNN language models (LM).

Word-based LMs display higher accuracy and lower computational cost than char-based LMs.

This drop of performance is unlikely due to the difficulty for character level model to capture longer short term memory, since also the Longer Short Term Memory (LSTM) recurrent networks work better with word-based input.

This is because char-based RNN LMs require much bigger hidden layer to successfully model long-term dependencies which means higher computational costs.

Therefore, we can say that

one of the fundamental differences between the word level and character level models is in the number of parameters the RNN has to access during the training and test. The smaller is the input and output layer of RNN, the larger needs to be the fully connected hidden layer, which makes the training of the model expensive.

However, char-based RNN LMs better model languages with a rich morphology such as Finish, Turkish, Russian etc. Using word-based RNN LMs to model such languages is difficult if possible at all and is not advised.

The above analysis makes sense especially when you look at the output text, generated by char-based RNNs:

The surprised in investors weren’t going to raise money. I’m not the company with the time there are all interesting quickly, don’t have to get off the same programmers.

While simple char-based Maximum Likelihood LM with a 13-character window delivers this:

And when she made many solid bricks. He stacked them in piles and stomped her feet. The doctor diagnosed him with a bat. The girl and her boyfriend asked her out.

Of course I cherry-picked the example (actually most ML LM examples looked better than any RNN generated text I've read so far) and this tiny ML LM was trained on a simpler corpus but you get the idea: straightforward conditional probability generates better texts than far more complex char-based RNN.

Char-based RNN LMs can mimic grammatically correct sequences for a wide range of languages, require bigger hidden layer and computationally more expensive while word-based RNN LMs train faster and generate more coherent texts and yet even these generated texts are far from making actual sense.

$\endgroup$
15
  • 1
    $\begingroup$ Excellent comment. It should be added that for some problems one or the other might make more sense regardless of computational concerns. For instance, if your goal is to study word vectors to find relations between words or if you want to generate a text based on a word-topic, then you have to go with word-based RNN. And, conversely, there probably are problems where char-based RNN is the way to go. It also depends on what the user is trying to do. $\endgroup$ Aug 5, 2016 at 4:30
  • 2
    $\begingroup$ @MinhKhôi this is just a logical conclusion after reading the linked paper: arxiv.org/abs/1511.06303 As for your confusion, you should be aware that word model PPL is not the same as char model PPL. $\endgroup$
    – minerals
    Jul 7, 2022 at 9:04
  • 1
    $\begingroup$ @MinhKhôi Since perplexity in LM is computed per token, it's meaningless to directly compare ppl on different tokenization. The relevant comparable is log(ppl-per-token)*token, aka $\log_2(44.9)$ * (num-of-words) vs (1.120)*(num-of-characters). It's clear that we can easily convert the ppl's using num-char-per-word $\endgroup$
    – shouldsee
    Aug 15, 2022 at 9:54
  • 1
    $\begingroup$ @MinhKhôi What is easier to represent and has shorter long term dependencies? "fat cat sat on a mat" = 6 units OR "f a t c a t s a t o n a m a t" = 15 units. Consider also that we deal with the English language, whose morphology is simple in comparison to German (where char-base models potentially could have an upper hand). $\endgroup$
    – minerals
    Aug 16, 2022 at 9:16
  • 1
    $\begingroup$ I agree with you, but my question about the test error is not really answered. You claimed that 'Word-based LMs display higher accuracy' and that 'char-based RNN LMs better model languages with a rich morphology such as Finish, Turkish, Russian, etc.', which does not align with my PTB benchmarks observation, and I just want to know why. $\endgroup$
    – Minh Khôi
    Aug 19, 2022 at 3:25
9
$\begingroup$

There is a nice write up about Language modeling a billion words. Below are some excerpts:

Word-level models have an important advantage over character-level models.
Take the following sequence as an example (a quote from Robert A. Heinlein):

Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something.

After tokenization, the word-level model might view this sequence as containing 22 tokens. On the other hand, the character-level will view this sequence as containing 102 tokens. This longer sequence makes the task of the character model harder than the word model, as it must take into account dependencies between more tokens over more time-steps. Another issue with character language models is that they need to learn spelling in addition to syntax, semantics, etc. In any case, word language models will typically have lower error than character models

The main advantage of character over word language models is that they have a really small vocabulary. For example, the GBW dataset will contain approximately 800 characters compared to 800,000 words (after pruning low-frequency tokens). In practice this means that character models will require less memory and have faster inference than their word counterparts. Another advantage is that they do not require tokenization as a preprocessing step.

$\endgroup$
2
$\begingroup$

In my opinion, the character based RNNs will also perform better but they need much more data than the word based models and character based models need to train for a much longer period of time. I would say it is more of a trial and error as well as a trade-off between data and computation power available.

$\endgroup$
0
$\begingroup$

Same: Both charRNN and wordRNN are auto-regressive models

Diff:

  1. CharRNN uses a smaller vocab table than WordRNN, with little OOV (out of vocabulary) occurrence. Whereas wordRNN needs to handle OOV explicitly to avoid zero probability. This also means wordRNN needs to handle vocab table carefully to avoid data leakage. (using words from test data only is essentially cheating.)
  2. CharRNN needs more autoregressive steps than wordRNN.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.