I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it.

Is there anything out-of-the box that I could use? *I tried to look for the sciBERT corpus, can not find how to access it.



Depends on the domain and language, but I'll share an adaptive example.

The wikipedia corpus's English version contains more than 1.9 billion words from 4.4 million articles.

You can create create virtual corpora from the full corpus to contain only topics of interest, such as biology, investments, Buddhism, psychology, cars, basketball, etc.

| improve this answer | |
  • $\begingroup$ Thanks! Do I need to do a web-crawl for that or is there a simpler way to do that that you can refer me to? * I don't have any experience with complicated methods for data mining, that's why I'm looking for "out of the box" corpus. $\endgroup$ – user Feb 20 at 15:46
  • $\begingroup$ @user Go to this link and here you can download the dataset. $\endgroup$ – userff Feb 20 at 15:53
  • $\begingroup$ I also used the reddit API a while back to get a 20M word corpus of jokes posted in 2019, if that's something that interests you. $\endgroup$ – userff Feb 20 at 15:54
  • $\begingroup$ I need a computer science related corpus, can not see there something like that, but I really appreciate you answers. $\endgroup$ – user Feb 21 at 1:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.