I have some experience with data science but I wanted some insight on how to deal with a very large dataset. I understand simply downloading it to your computer is not plausible so where do you even begin? Say for example I want to download a particular dataset from some website that is over 10gb.
5
-
2$\begingroup$ You might need to specify what kind of data you are working with. What considered big is different for different tasks in data science. 10 gb of image is relatively small or at most medium size but if your data is tabular (e.g. customer transaction data) the dataset can be considered huge. There are many methods to deal with large datasets but different tasks require different types of treatement. $\endgroup$– Yohanes AlfredoCommented Nov 21, 2019 at 8:11
-
$\begingroup$ Too broad: please be more specific regarding the type of data and what you want to do with it. 10 GB is not "large" btw. $\endgroup$– PeterCommented Nov 21, 2019 at 9:21
-
$\begingroup$ I agree with the above two comments, you should be more specific when you ask a question. I worked on a lung CT scan image dataset that has 120 GB of size. $\endgroup$– HunarCommented Nov 21, 2019 at 10:08
-
$\begingroup$ Sorry about that. Specifically for image data, what would an approach be? I want to do some practice with image segmentation. $\endgroup$– jeffCommented Nov 21, 2019 at 18:52
-
$\begingroup$ Does this answer your question? When a dataset is huge, what do you do to train with all the images on i t? $\endgroup$– PeterCommented Aug 23, 2020 at 9:12
Add a comment
|