2
$\begingroup$

for a training purposes, in order to start machine learning and data analysis with python I work on a pretty small dataset (20000 rows, 600MB) .But since couple days I decided to go further so I have download a large data set of 1Go.

I wanted to do some analysis and apply machine learning on it so I tried to read the csv file with pyhton in jupiter nootebook and th file is still loading after more than 3 hours.

So I will would like to know what are the best practices/process to follow when you have to work with large dataset ?

$\endgroup$
3
$\begingroup$

Before exploring more sophisticated tools like Spark or Dask, one option would be to read the data in chunks instead of loading the whole file. For example, if you are using pandas the read_csv method accepts chunksize argument.

The main idea is that often what you need to do is reduce each chunk down to something much smaller with only the parts you need (some average/count/sum etc.. of some classes) and store a series/array/dict etc accordingly.

$\endgroup$
2
$\begingroup$

Keras

If you are using Keras, there is a class Sequence which generates data. You need to implement the way how data will be generated. In your case, you read a file for a couple of lines and then return those lines.

This way, you will be able to use the data faster because you don't need to read all the data (whole .csv file) at once. This is usually called reading data in chunks.

Same data reading procedure can be achieved also in other frameworks (TF, Pytorch, ...).

Pure Python

If you really want to do read data in chunks in pure Python, you could use yield statement in Python. More about yield and generators can be found here and here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.