Timeline for What is the difference between word-based and char-based text generation RNNs?
Current License: CC BY-SA 3.0
21 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 26, 2022 at 7:51 | comment | added | shouldsee | @minerals. Thanks for sharing. Tho I guess it would be easier to compare quantitative evidence than empirical evidence.... People just love to cherry-interpret empirical evidence from all different aspects :D | |
| Aug 19, 2022 at 3:25 | comment | added | Minh Khôi | I agree with you, but my question about the test error is not really answered. You claimed that 'Word-based LMs display higher accuracy' and that 'char-based RNN LMs better model languages with a rich morphology such as Finish, Turkish, Russian, etc.', which does not align with my PTB benchmarks observation, and I just want to know why. | |
| Aug 18, 2022 at 11:32 | comment | added | tastyminerals | @MinhKhôi It's what you are trying to model. Word-based models learn the dependencies between words, char-based models learn the dependencies between morphemes. Whether char-based models are able to model the syntactical dependencies is rather a question of data amount, representation and type of model used. | |
| Aug 17, 2022 at 11:02 | comment | added | Minh Khôi | Shorter long-term dependencies do not always imply a better convergence rate or generalization error. Do you mean that it depends on the LM and the language being considered? | |
| Aug 16, 2022 at 9:16 | comment | added | tastyminerals | @MinhKhôi What is easier to represent and has shorter long term dependencies? "fat cat sat on a mat" = 6 units OR "f a t c a t s a t o n a m a t" = 15 units. Consider also that we deal with the English language, whose morphology is simple in comparison to German (where char-base models potentially could have an upper hand). | |
| Aug 16, 2022 at 9:01 | comment | added | tastyminerals | @shouldsee not a quantitative but empirical ;) see the examples generated by a toy ML model here: github.com/tastyminerals/char-ml-generator | |
| Aug 16, 2022 at 6:07 | comment | added | Minh Khôi | @shouldsee: I agree that we should not compare PPLs of different tokenization. Assuming the accuracies are also calculated differently with different tokenization. Then in what sense did minerals mean that the Word-based LM display higher accuracy than Char-based LM? | |
| Aug 15, 2022 at 10:09 | comment | added | shouldsee | ` (actually most ML LM examples looked better than any RNN generated text I've read so far)`. Do you have quantitative evidence to support this bold claim? (doge) | |
| Aug 15, 2022 at 9:54 | comment | added | shouldsee | @MinhKhôi Since perplexity in LM is computed per token, it's meaningless to directly compare ppl on different tokenization. The relevant comparable is log(ppl-per-token)*token, aka $\log_2(44.9)$ * (num-of-words) vs (1.120)*(num-of-characters). It's clear that we can easily convert the ppl's using num-char-per-word | |
| Jul 7, 2022 at 9:04 | comment | added | tastyminerals | @MinhKhôi this is just a logical conclusion after reading the linked paper: arxiv.org/abs/1511.06303 As for your confusion, you should be aware that word model PPL is not the same as char model PPL. | |
| Jun 23, 2022 at 9:38 | comment | added | Minh Khôi | @minerals: "Word-based LMs display higher accuracy and lower computational cost than char-based LMs." Do you have a source for this one? As ranked on paperswithcode.com, the dataset Penn Treebank has the current best RNN-related word modeling perplexity at $44.9$ while the character modeling task has the best 1.120 BPC. These resulted in the respective Cross-Entropy loss of $3.8$ and $0.776$, which greatly confused me. It'd help if you could set some clear clarification regarding this matter. | |
| Dec 23, 2017 at 19:28 | comment | added | Claude COULOMBE | Great answer! Indeed someone could easily add that it depends a lot on the task involved, the size of your dataset, the languages and the level of pre-processing you're willing to do. For example, to process richer morphology languages and to manage out of vocabulary (OOV) word, you can also use word-model with lemmatization, pos tagging, and add prefixes, suffixes, etc. | |
| Aug 13, 2016 at 11:32 | vote | accept | tastyminerals | ||
| Aug 5, 2016 at 11:43 | history | edited | tastyminerals | CC BY-SA 3.0 |
added 74 characters in body
|
| Aug 5, 2016 at 11:38 | history | edited | tastyminerals | CC BY-SA 3.0 |
added 74 characters in body
|
| Aug 5, 2016 at 11:33 | comment | added | tastyminerals | I've updated the ambiguous ending. | |
| Aug 5, 2016 at 11:33 | history | edited | tastyminerals | CC BY-SA 3.0 |
added 74 characters in body
|
| Aug 5, 2016 at 11:25 | vote | accept | tastyminerals | ||
| Aug 5, 2016 at 11:25 | |||||
| Aug 5, 2016 at 4:30 | comment | added | Ricardo Cruz | I did not understand your last comment: "Char-based RNN LM (...) fall short when it comes to making actual sense." I haven't seen a Word-based RNN making sense either. Why did you isolate char-based models here? | |
| Aug 5, 2016 at 4:30 | comment | added | Ricardo Cruz | Excellent comment. It should be added that for some problems one or the other might make more sense regardless of computational concerns. For instance, if your goal is to study word vectors to find relations between words or if you want to generate a text based on a word-topic, then you have to go with word-based RNN. And, conversely, there probably are problems where char-based RNN is the way to go. It also depends on what the user is trying to do. | |
| Aug 3, 2016 at 20:30 | history | answered | tastyminerals | CC BY-SA 3.0 |