I have some experience with data science but I wanted some insight on how to deal with a very large dataset. I understand simply downloading it to your computer is not plausible so where do you even begin? Say for example I want to download a particular dataset from some website that is over 10gb.

  • 2
    $\begingroup$ You might need to specify what kind of data you are working with. What considered big is different for different tasks in data science. 10 gb of image is relatively small or at most medium size but if your data is tabular (e.g. customer transaction data) the dataset can be considered huge. There are many methods to deal with large datasets but different tasks require different types of treatement. $\endgroup$ Nov 21 '19 at 8:11
  • $\begingroup$ Too broad: please be more specific regarding the type of data and what you want to do with it. 10 GB is not "large" btw. $\endgroup$
    – Peter
    Nov 21 '19 at 9:21
  • $\begingroup$ I agree with the above two comments, you should be more specific when you ask a question. I worked on a lung CT scan image dataset that has 120 GB of size. $\endgroup$
    – Hunar
    Nov 21 '19 at 10:08
  • $\begingroup$ Sorry about that. Specifically for image data, what would an approach be? I want to do some practice with image segmentation. $\endgroup$
    – jeff
    Nov 21 '19 at 18:52
  • $\begingroup$ Does this answer your question? When a dataset is huge, what do you do to train with all the images on i t? $\endgroup$
    – Peter
    Aug 23 '20 at 9:12