How to load a csv file into [Pandas] dataframe if computer runs out of RAM?

Question

I have been trying to train a neural network, but my computer is always running out of RAM memory when I'm loading the dataframe with Pandas. Its a .csv file that is like 7+ GB.

I wanted to try some primitive batching but in order to one hot encode I need to find number of all unique values, which i can't do without loading data into a dataframe first.

Are there any other tools that I can use to attempt loading the file into a dataframe? Does Pyspark also have a limit of when it starts crashing? I know that its capable of breaking down operations into stages, does that help with RAM management or just execution?

Answer 1

As noted many times by the writers of pandas, the ideal size of memory for analyzing with pandas is around 5-10 times the load of data you are giving.

That being said, if you can afford to load data is chunks for pre-processing and dump only the columns that are needed, I recommended pandas.read_table option to load data in pieces (check option chunks).

pyspark as you mentioned might be good to go with. But there is dask which is built around python stack of pandas and numpy for distributed work loads which has support for pandas.DataFrame and numpy.array . But I never followed anyone who was successful in using it in production (or atleast mentioned that they had used it). May be some of the people here can vouch for it. There is another library called sframe to load the data way out of memory as the author suggested in one of his presentations. It keeps on serializing data onto disk.

Hope I summarized it well.