I am creating a tool that reads CSV fields and allows the user to specify columns they want to categorize and then categorize those columns.

My issue is that these CSV files are quite large and when trying to concatenate the data-frames, my PC freezes up and I get a MemoryError.

I split the data-frame into chunks and complete the get_dummies function on each chunk and store it into a list. This works without any issues.

I then try to concatenate the entire list, as you can see in the code below.

I also delete the data-frames and the list of chunks to save up on memory.

dummies = []
columns = self.df[self.selectedHeaders]
del self.df
chunks = (len(columns) / 10000) + 1
df_list = np.array_split(columns, chunks)
del columns

for i, df_chunk in enumerate(df_list):
    print("Getting dummy data for chunk: " + str(i))

del df_list
dummies = pd.concat(dummies, axis=1)

As you can see from this code, I store the columns I need and split them into chunks. Then I run the get_dummies function on each chunk and store them within a list.

When I run the concat function, I either crash or a MemoryError. If I can get the code to run and throw that error without crashing, I'll update it here.

  • $\begingroup$ I've been there. And the only thing that I have learned is to avoid pandas for large datasets. It just doesn't scale. scikit-learn provides plenty of highly optimized data structures. I would recommend using them. $\endgroup$
    – sandyp
    Oct 24, 2019 at 17:58
  • $\begingroup$ You can try dask. It will probably be able to handle bigger datasets $\endgroup$
    – Tasos
    Nov 3, 2019 at 11:52

1 Answer 1


You should use sparse matrices

When you are categorizing a column, you are creating many columns that mostly hold zeroes, and since dense data structures keep track of all the zeros, the memory usage explodes. Sparse matrices on the other hand only keep track of non-zero values and their indexes. There are many version of sparse matrices optimized for different use cases.

  • $\begingroup$ Which one do you recommend for One-Hot-Encoding? $\endgroup$
    – Top Lit
    Oct 25, 2019 at 6:46
  • $\begingroup$ @TopLit, it depends what you will do with the matrix, I recently constructed a coo_matrix because it is easy to construct and passed to to sklearn model, which I believe converts everything to csr_matrix for fast matrix operations. Each sparse matrix type has its intended usage, and it is not uncommon to switch between types when your usage changes. The advantages and disadvantages are provided in the documentation. $\endgroup$
    – Akavall
    Oct 25, 2019 at 15:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.