Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I use pandas.read_csv to read a huge file for machine learning, but I got memory error.
Someone recommend me to set arg chunksize but I need sort, random access...etc. So I need to load whole data into memory or use another way.
Some ways I think it's possible is Hadoop. Another one is incremental training, but it's like reading chunksize in read_csv
Or other software/library/ways can I use?
I would suggest you to use Dask. I used it successfully when I had to read large data with my 4GB RAM. You can get more details here.
To read a CSV, you can do the following:
import dask.dataframe as dd
csv_file = 'data.csv'
df = dd.read_csv(csv_file)
