# What’s the best way to save many pandas dataframes together?

I’m looking for a way to save house prices data by city, for example a pandas panel with one dataframe per city. But I need the dataframes to be independent, meaning that if one dataframe is corrupted, the others are untouched. I tried using pickle and csv, but once a line was corrupted I lost the whole file.

• Save it in feather format... Also if you are corrupting any line, you are changing a bit of data in CS which will obviously lead to corruption of your data, So keep it in a seperate directory all together.. – Aditya Jun 15 '18 at 4:21
• Save it in any format except Pickle. Pickle is absurdly fragile. – Stephen Rauch Jun 15 '18 at 4:50
• @StephenRauch - I don't doubt it, but I was wondering what makes you say pickle is so fragile? I know about the versions and backwards compatibility constraints, but is there anything else I should know? – n1k31t4 Jun 15 '18 at 21:40
• @n1k31t4, the fundamental problem is as you mentioned. These things make the strorage format fragile in a production environment. – Stephen Rauch Jun 16 '18 at 0:02

If you want to get quite involved and be able to specify names for each of the panels you create, you could look at the h5 file format.

This allows you to group datasets in named containers. You can then read them from disk later on one by one i.e. you don't need to read the whole dataset into memory.

Here is an example of a function that would save such a dataset:

def save_h5(h5_filename, data, labels, descr=None,
data_dtype='float32', label_dtype='float32'):
"""Create a compressed .h5 file containing:
data    : numpy array
labels  : numpy array
descr   : text description ofthe data contained (must be a string)
"""

if os.path.exists(h5_filename):
# prevent overwriting a file

h5_fout = h5py.File(h5_filename)

h5_fout.create_dataset(
name='data',
data=data,
compression='gzip', compression_opts=4,
dtype=data_dtype)

h5_fout.create_dataset(
name='labels',
data=labels,
compression='gzip', compression_opts=4,
dtype=label_dtype)

if descr is not None:
h5_fout.create_dataset(
'description', data=descr)

h5_fout.close()


For the meaning of the parameters, head over the documentation.

You can write a similar function to make accessing the saved h5 file. This really is a flexible way to save data, and it can be compressed with one of the best known (widely-spread) algorithms in the open-source world: gzip! There are also other possibilities implemented.

On a side note, if you want to minimise the possibility of corruption, you could consider saving each panel/DataFrame (whichever method you go for) into separate files, and then make copies/backups.

... the beauty of a simple csv file is that you can actually open it in notepad or a spreadsheet and usually find the line which is "broken" and fix/delete it. Pickle, on the other hand, is a little more complicated to debug.
• That's weird that is it still showing 1Gb on disk. Can you measure the size with import os; os.path.getsize('path/to/file.csv')? If you're on Windows, it might be worth trying [this] (support.microsoft.com/en-us/help/17119/…) – n1k31t4 Jun 15 '18 at 18:04