# MemoryError when concatenating a large data-frame

I am creating a tool that reads CSV fields and allows the user to specify columns they want to categorize and then categorize those columns.

My issue is that these CSV files are quite large and when trying to concatenate the data-frames, my PC freezes up and I get a MemoryError.

I split the data-frame into chunks and complete the get_dummies function on each chunk and store it into a list. This works without any issues.

I then try to concatenate the entire list, as you can see in the code below.

I also delete the data-frames and the list of chunks to save up on memory.

dummies = []
del self.df
chunks = (len(columns) / 10000) + 1
df_list = np.array_split(columns, chunks)
del columns

for i, df_chunk in enumerate(df_list):
print("Getting dummy data for chunk: " + str(i))
dummies.append(pd.get_dummies(df_chunk))

del df_list
dummies = pd.concat(dummies, axis=1)


As you can see from this code, I store the columns I need and split them into chunks. Then I run the get_dummies function on each chunk and store them within a list.

When I run the concat function, I either crash or a MemoryError. If I can get the code to run and throw that error without crashing, I'll update it here.

• I've been there. And the only thing that I have learned is to avoid pandas for large datasets. It just doesn't scale. scikit-learn provides plenty of highly optimized data structures. I would recommend using them. – sandyp Oct 24 '19 at 17:58
• You can try dask. It will probably be able to handle bigger datasets – Tasos Nov 3 '19 at 11:52

• @TopLit, it depends what you will do with the matrix, I recently constructed a coo_matrix because it is easy to construct and passed to to sklearn model, which I believe converts everything to csr_matrix for fast matrix operations. Each sparse matrix type has its intended usage, and it is not uncommon to switch between types when your usage changes. The advantages and disadvantages are provided in the documentation. – Akavall Oct 25 '19 at 15:27