I have some experience with data science but I wanted some insight on how to deal with a very large dataset. I understand simply downloading it to your computer is not plausible so where do you even begin? Say for example I want to download a particular dataset from some website that is over 10gb.

  • 2
    $\begingroup$ You might need to specify what kind of data you are working with. What considered big is different for different tasks in data science. 10 gb of image is relatively small or at most medium size but if your data is tabular (e.g. customer transaction data) the dataset can be considered huge. There are many methods to deal with large datasets but different tasks require different types of treatement. $\endgroup$ – Yohanes Alfredo Nov 21 '19 at 8:11
  • $\begingroup$ Too broad: please be more specific regarding the type of data and what you want to do with it. 10 GB is not "large" btw. $\endgroup$ – Peter Nov 21 '19 at 9:21
  • $\begingroup$ I agree with the above two comments, you should be more specific when you ask a question. I worked on a lung CT scan image dataset that has 120 GB of size. $\endgroup$ – Hunar A.Ahmed Nov 21 '19 at 10:08
  • $\begingroup$ Sorry about that. Specifically for image data, what would an approach be? I want to do some practice with image segmentation. $\endgroup$ – jeff Nov 21 '19 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.