1
$\begingroup$

I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it.

Is there anything out-of-the box that I could use? *I tried to look for the sciBERT corpus, can not find how to access it.

Thanks!

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
0
$\begingroup$

Depends on the domain and language, but I'll share an adaptive example.

The wikipedia corpus's English version contains more than 1.9 billion words from 4.4 million articles.

You can create create virtual corpora from the full corpus to contain only topics of interest, such as biology, investments, Buddhism, psychology, cars, basketball, etc.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks! Do I need to do a web-crawl for that or is there a simpler way to do that that you can refer me to? * I don't have any experience with complicated methods for data mining, that's why I'm looking for "out of the box" corpus. $\endgroup$
    – user
    Feb 20, 2020 at 15:46
  • $\begingroup$ @user Go to this link and here you can download the dataset. $\endgroup$
    – gust
    Feb 20, 2020 at 15:53
  • $\begingroup$ I also used the reddit API a while back to get a 20M word corpus of jokes posted in 2019, if that's something that interests you. $\endgroup$
    – gust
    Feb 20, 2020 at 15:54
  • $\begingroup$ I need a computer science related corpus, can not see there something like that, but I really appreciate you answers. $\endgroup$
    – user
    Feb 21, 2020 at 1:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.