As the toronto book corpus is no longer available (or rather, only in lowercase), I am looking for an alternative dataset of comparable language variety, quality, and size. Any suggestions? The Gutenberg Standardized Corpus is too big and still requires lots of preprocessing.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
First, for context, I suggest other bypasses check the writeup of a researcher who also tried to find the Toronto Book Corpus.
There is a potential copy of the corpus shared by Igor Brigadir on Twitter , although it is not certain that it is the same exact corpus (see discussion). HuggingFace datasets hosts a copy of this corpus. As you noted, this version is in lowercase.
There are other people who have replicated the corpus to some degree, like Shawn Presser, who shared it on Twitter (download link). Here is some context for this replication and more info on the matter. This replication is NOT in lowercase.
Also, here you can find the instructions and code to replicate it yourself.
Finally, there is this paper studying the problems of the Toronto Book Corpus and its replications.
-
$\begingroup$ Thank you for your answer. I am using the huggingface version, but it's in lowercase to my dismay. That is why I was asking for alternatives. Your answer may be still helpful for people looking for the corpus regardless of casing. $\endgroup$ Commented Feb 13, 2023 at 12:54
-
1$\begingroup$ Note that the replication by Shawn Presser is not in lowercase. $\endgroup$– noeCommented Feb 13, 2023 at 12:57
-
1