MemoryError when concatenating a large data-frame

Question

I am creating a tool that reads CSV fields and allows the user to specify columns they want to categorize and then categorize those columns.

My issue is that these CSV files are quite large and when trying to concatenate the data-frames, my PC freezes up and I get a MemoryError.

I split the data-frame into chunks and complete the get_dummies function on each chunk and store it into a list. This works without any issues.

I then try to concatenate the entire list, as you can see in the code below.

I also delete the data-frames and the list of chunks to save up on memory.

dummies = []
columns = self.df[self.selectedHeaders]
del self.df
chunks = (len(columns) / 10000) + 1
df_list = np.array_split(columns, chunks)
del columns

for i, df_chunk in enumerate(df_list):
    print("Getting dummy data for chunk: " + str(i))
    dummies.append(pd.get_dummies(df_chunk))

del df_list
dummies = pd.concat(dummies, axis=1)

As you can see from this code, I store the columns I need and split them into chunks. Then I run the get_dummies function on each chunk and store them within a list.

When I run the concat function, I either crash or a MemoryError. If I can get the code to run and throw that error without crashing, I'll update it here.

Answer 1

You should use sparse matrices

When you are categorizing a column, you are creating many columns that mostly hold zeroes, and since dense data structures keep track of all the zeros, the memory usage explodes. Sparse matrices on the other hand only keep track of non-zero values and their indexes. There are many version of sparse matrices optimized for different use cases.