I am creating a tool that reads CSV fields and allows the user to specify columns they want to categorize and then categorize those columns.
My issue is that these CSV files are quite large and when trying to concatenate the data-frames, my PC freezes up and I get a MemoryError.
I split the data-frame into chunks and complete the
get_dummies function on each chunk and store it into a list. This works without any issues.
I then try to concatenate the entire list, as you can see in the code below.
I also delete the data-frames and the list of chunks to save up on memory.
dummies =  columns = self.df[self.selectedHeaders] del self.df chunks = (len(columns) / 10000) + 1 df_list = np.array_split(columns, chunks) del columns for i, df_chunk in enumerate(df_list): print("Getting dummy data for chunk: " + str(i)) dummies.append(pd.get_dummies(df_chunk)) del df_list dummies = pd.concat(dummies, axis=1)
As you can see from this code, I store the columns I need and split them into chunks. Then I run the
get_dummies function on each chunk and store them within a list.
When I run the
concat function, I either crash or a MemoryError. If I can get the code to run and throw that error without crashing, I'll update it here.