I use pandas.read_csv
to read a huge file for machine learning, but I got memory error.
Someone recommend me to set arg chunksize
but I need sort, random access...etc. So I need to load whole data into memory or use another way.
Some ways I think it's possible is Hadoop. Another one is incremental training, but it's like reading chunksize in read_csv
Or other software/library/ways can I use?