15
$\begingroup$

While reading about text generation with Recurrent Neural Networks I noticed that some examples were implemented to generate text word by word and others character by character without actually stating why.

So, what is the difference between RNN models that predict text per-word basis and the ones that predict text per-char basis? Do word-based RNN require a bigger corpus size? Do char-based RNN generalize better? Maybe the only difference is input representation (one-hot encoding, word embeddings)? Which ones to choose for text generation?

$\endgroup$
15
$\begingroup$

Here is what I learnt recently.

Obviously, when talking about text generation RNNs we are talking about RNN language models. When asking about word/char-based text generation RNNs, we are asking about word/char-based RNN language models (LM).

Word-based LMs display higher accuracy and lower computational cost than char-based LMs.

This drop of performance is unlikely due to the difficulty for character level model to capture longer short term memory, since also the Longer Short Term Memory (LSTM) recurrent networks work better with word-based input.

This is because char-based RNN LMs require much bigger hidden layer to successfully model long-term dependencies which means higher computational costs.

Therefore, we can say that

one of the fundamental differences between the word level and character level models is in the number of parameters the RNN has to access during the training and test. The smaller is the input and output layer of RNN, the larger needs to be the fully connected hidden layer, which makes the training of the model expensive.

However, char-based RNN LMs better model languages with a rich morphology such as Finish, Turkish, Russian etc. Using word-based RNN LMs to model such languages is difficult if possible at all and is not advised.

The above analysis makes sense especially when you look at the output text, generated by char-based RNNs:

The surprised in investors weren’t going to raise money. I’m not the company with the time there are all interesting quickly, don’t have to get off the same programmers.

While simple char-based Maximum Likelihood LM with a 13-character window delivers this:

And when she made many solid bricks. He stacked them in piles and stomped her feet. The doctor diagnosed him with a bat. The girl and her boyfriend asked her out.

Of course I cherry-picked the example (actually most ML LM examples looked better than any RNN generated text I've read so far) and this tiny ML LM was trained on a simpler corpus but you get the idea: straightforward conditional probability generates better texts than far more complex char-based RNN.

Char-based RNN LMs can mimic grammatically correct sequences for a wide range of languages, require bigger hidden layer and computationally more expensive while word-based RNN LMs train faster and generate more coherent texts and yet even these generated texts are far from making actual sense.

$\endgroup$
  • 1
    $\begingroup$ Excellent comment. It should be added that for some problems one or the other might make more sense regardless of computational concerns. For instance, if your goal is to study word vectors to find relations between words or if you want to generate a text based on a word-topic, then you have to go with word-based RNN. And, conversely, there probably are problems where char-based RNN is the way to go. It also depends on what the user is trying to do. $\endgroup$ – Ricardo Cruz Aug 5 '16 at 4:30
  • $\begingroup$ I did not understand your last comment: "Char-based RNN LM (...) fall short when it comes to making actual sense." I haven't seen a Word-based RNN making sense either. Why did you isolate char-based models here? $\endgroup$ – Ricardo Cruz Aug 5 '16 at 4:30
  • $\begingroup$ I've updated the ambiguous ending. $\endgroup$ – minerals Aug 5 '16 at 11:33
  • $\begingroup$ Great answer! Indeed someone could easily add that it depends a lot on the task involved, the size of your dataset, the languages and the level of pre-processing you're willing to do. For example, to process richer morphology languages and to manage out of vocabulary (OOV) word, you can also use word-model with lemmatization, pos tagging, and add prefixes, suffixes, etc. $\endgroup$ – Claude COULOMBE Dec 23 '17 at 19:28
5
$\begingroup$

There is a nice write up about Language modeling a billion words. Below are some excerpts:

Word-level models have an important advantage over character-level models.
Take the following sequence as an example (a quote from Robert A. Heinlein):

Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something.

After tokenization, the word-level model might view this sequence as containing 22 tokens. On the other hand, the character-level will view this sequence as containing 102 tokens. This longer sequence makes the task of the character model harder than the word model, as it must take into account dependencies between more tokens over more time-steps. Another issue with character language models is that they need to learn spelling in addition to syntax, semantics, etc. In any case, word language models will typically have lower error than character models

The main advantage of character over word language models is that they have a really small vocabulary. For example, the GBW dataset will contain approximately 800 characters compared to 800,000 words (after pruning low-frequency tokens). In practice this means that character models will require less memory and have faster inference than their word counterparts. Another advantage is that they do not require tokenization as a preprocessing step.

$\endgroup$
1
$\begingroup$

In my opinion, the character based RNNs will also perform better but they need much more data than the word based models and character based models need to train for a much longer period of time. I would say it is more of a trial and error as well as a trade-off between data and computation power available.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.