I am currently doing a project which requires a dataset. May I know whether there are any free large public datasets that has the following characteristics:
- greater than 3GB
- non-tabular (each txt file contains a weblog/ written paragraph)
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
You can collect research articles. A large subset of the biomedical literature can be downloaded from PubMed:
- Medline is the full collection of abstracts (around 30 millions).
- PubMedCentral is a subset of around 3 millions full papers (as opposed to containing only the abstract).
Both are very large, more than 50GB as far as I remember.
$\begingroup$
$\endgroup$
You can use Kaggle and filter all data sets that are greater than 3Gb and that have text content.
The data is generally speaking free, but a subscription is needed and there are some rules to follow.
For instance: https://www.kaggle.com/datasets?sizeStart=3000%2CMB&tags=14104-Text+Data
answered Jul 4, 2022 at 7:33
$\begingroup$
$\endgroup$
2
Two sources which have not been mentioned yet are Huggingface and Papers with code. While you cannot filter for dataset disk space, Huggingface allows filtering for dataset size in terms of datapoints/rows. The English Wikipedia dataset, for example, is about 20GB (and requires 40GB disk space).
-
1$\begingroup$ Minor correction: according to that page Wikipedia (20220301.en) is 20GB (and requires 40GB disk space). $\endgroup$ Jul 4, 2022 at 20:56
-
$\begingroup$ Thanks, @DarrenCook - I have edited my answer accordingly. $\endgroup$– JonathanJul 11, 2022 at 10:52