1
$\begingroup$

I use pandas.read_csv to read a huge file for machine learning, but I got memory error.

Someone recommend me to set arg chunksize but I need sort, random access...etc. So I need to load whole data into memory or use another way.

Some ways I think it's possible is Hadoop. Another one is incremental training, but it's like reading chunksize in read_csv

Or other software/library/ways can I use?

| improve this question | | | | |
$\endgroup$
  • 1
    $\begingroup$ Check this out $\endgroup$ – Kiritee Gak Mar 19 '19 at 8:43
2
$\begingroup$

I would suggest you to use Dask. I used it successfully when I had to read large data with my 4GB RAM. You can get more details here.

To read a CSV, you can do the following:

import dask.dataframe as dd

csv_file = 'data.csv'
df = dd.read_csv(csv_file)
| improve this answer | | | | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.