1
$\begingroup$

I use pandas.read_csv to read a huge file for machine learning, but I got memory error.

Someone recommend me to set arg chunksize but I need sort, random access...etc. So I need to load whole data into memory or use another way.

Some ways I think it's possible is Hadoop. Another one is incremental training, but it's like reading chunksize in read_csv

Or other software/library/ways can I use?

$\endgroup$
1
  • 1
    $\begingroup$ Check this out $\endgroup$ Mar 19, 2019 at 8:43

1 Answer 1

2
$\begingroup$

I would suggest you to use Dask. I used it successfully when I had to read large data with my 4GB RAM. You can get more details here.

To read a CSV, you can do the following:

import dask.dataframe as dd

csv_file = 'data.csv'
df = dd.read_csv(csv_file)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.