1
$\begingroup$

I am currently doing a project which requires a dataset. May I know whether there are any free large public datasets that has the following characteristics:

  1. greater than 3GB
  2. non-tabular (each txt file contains a weblog/ written paragraph)
$\endgroup$

3 Answers 3

1
$\begingroup$

You can collect research articles. A large subset of the biomedical literature can be downloaded from PubMed:

  • Medline is the full collection of abstracts (around 30 millions).
  • PubMedCentral is a subset of around 3 millions full papers (as opposed to containing only the abstract).

Both are very large, more than 50GB as far as I remember.

$\endgroup$
1
$\begingroup$

You can use Kaggle and filter all data sets that are greater than 3Gb and that have text content.

The data is generally speaking free, but a subscription is needed and there are some rules to follow.

For instance: https://www.kaggle.com/datasets?sizeStart=3000%2CMB&tags=14104-Text+Data

$\endgroup$
1
$\begingroup$

Two sources which have not been mentioned yet are Huggingface and Papers with code. While you cannot filter for dataset disk space, Huggingface allows filtering for dataset size in terms of datapoints/rows. The English Wikipedia dataset, for example, is about 20GB (and requires 40GB disk space).

$\endgroup$
2
  • 1
    $\begingroup$ Minor correction: according to that page Wikipedia (20220301.en) is 20GB (and requires 40GB disk space). $\endgroup$ Commented Jul 4, 2022 at 20:56
  • $\begingroup$ Thanks, @DarrenCook - I have edited my answer accordingly. $\endgroup$
    – Jonathan
    Commented Jul 11, 2022 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.