I’m looking for a way to save house prices data by city, for example a pandas panel with one dataframe per city. But I need the dataframes to be independent, meaning that if one dataframe is corrupted, the others are untouched. I tried using pickle and csv, but once a line was corrupted I lost the whole file.
If you want to get quite involved and be able to specify names for each of the panels you create, you could look at the h5 file format.
This allows you to group datasets in named containers. You can then read them from disk later on one by one i.e. you don't need to read the whole dataset into memory.
Here is an example of a function that would save such a dataset:
def save_h5(h5_filename, data, labels, descr=None, data_dtype='float32', label_dtype='float32'): """Create a compressed .h5 file containing: data : numpy array labels : numpy array descr : text description ofthe data contained (must be a string) """ if os.path.exists(h5_filename): # prevent overwriting a file sys.exit('File already exists!') h5_fout = h5py.File(h5_filename) h5_fout.create_dataset( name='data', data=data, compression='gzip', compression_opts=4, dtype=data_dtype) h5_fout.create_dataset( name='labels', data=labels, compression='gzip', compression_opts=4, dtype=label_dtype) if descr is not None: h5_fout.create_dataset( 'description', data=descr) h5_fout.close()
For the meaning of the parameters, head over the documentation.
You can write a similar function to make accessing the saved h5 file. This really is a flexible way to save data, and it can be compressed with one of the best known (widely-spread) algorithms in the open-source world: gzip! There are also other possibilities implemented.
On a side note, if you want to minimise the possibility of corruption, you could consider saving each panel/DataFrame (whichever method you go for) into separate files, and then make copies/backups.
Additionally, you said:
I tried using pickle and csv, but once a line was corrupted I lost the whole file.
... the beauty of a simple
csv file is that you can actually open it in notepad or a spreadsheet and usually find the line which is "broken" and fix/delete it. Pickle, on the other hand, is a little more complicated to debug.