I've observed that many of the datasets available for traditional ML and data science algorithms seem to be in the order of MB. I assumed these may be because earlier computers were not that computationally efficient. However, someone pointed out it need not be the case. Many of the training datasets are smaller in size.

I want to ask whether or not using a huge dataset for training/inference on traditional ML and data science algorithms makes sense? Is it a proper use case? If yes, can someone please point me towards such a huge dataset available online?

  • $\begingroup$ Welcome to DataScienceSE. The question is either too broad or ambiguous: either it's specifically about business intelligence datasets (I assume such open datasets are rare because they are valuable/sensitive), or it's about general data used for ML, in which case the premise is wrong. For example the Medline corpus contains 30 millions abstract, it's around 200GB of text. If you're interested in finding open data about a specific topic you can try asking on opendata.stackexchange.com. $\endgroup$
    – Erwan
    Commented May 7, 2021 at 12:59
  • $\begingroup$ Thank you. I just wanted to make a case for why these algorithms should be processed on GPUs rather than only CPUs. I know there are certain libraries that tend to do that, but I just wanted to make sure whether this qualifies as a valid test case. For example, an invalid test case would be to perform Deep learning training on mobile phones... that makes no sense. Similarly, if these algorithms are good enough for CPUs, there is no need to process them on GPUs. $\endgroup$ Commented May 7, 2021 at 13:56
  • $\begingroup$ Sure, in general a small dataset doesn't require GPU speed acceleration (although that might also depend on the complexity of model). But it's still not clear to me what is your question? $\endgroup$
    – Erwan
    Commented May 7, 2021 at 22:01
  • $\begingroup$ Thanks for the response. I am new to this field and was curious to notice that many of the available datasets for traditional ML (not deep learning) have smaller sizes and CPUs seem to be a natural choice. However, there are certain libraries RAPIDS, for example, which try to implement these on GPUs. I was under the assumption that the traditional ML models almost always work on smaller datasets (unlike deep learning) and thus whether training them on GPUs was a good use case or not. That's why I asked whether businesses use very large datasets, which might make the case for GPU acceleration. $\endgroup$ Commented May 7, 2021 at 22:10


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.