I’m looking for a way to save house prices data by city, for example a pandas panel with one dataframe per city. But I need the dataframes to be independent, meaning that if one dataframe is corrupted, the others are untouched. I tried using pickle and csv, but once a line was corrupted I lost the whole file.

  • $\begingroup$ Save it in feather format... Also if you are corrupting any line, you are changing a bit of data in CS which will obviously lead to corruption of your data, So keep it in a seperate directory all together.. $\endgroup$
    – Aditya
    Jun 15, 2018 at 4:21
  • $\begingroup$ Save it in any format except Pickle. Pickle is absurdly fragile. $\endgroup$
    – Stephen Rauch
    Jun 15, 2018 at 4:50
  • $\begingroup$ @StephenRauch - I don't doubt it, but I was wondering what makes you say pickle is so fragile? I know about the versions and backwards compatibility constraints, but is there anything else I should know? $\endgroup$
    – n1k31t4
    Jun 15, 2018 at 21:40
  • $\begingroup$ @n1k31t4, the fundamental problem is as you mentioned. These things make the strorage format fragile in a production environment. $\endgroup$
    – Stephen Rauch
    Jun 16, 2018 at 0:02

1 Answer 1


If you want to get quite involved and be able to specify names for each of the panels you create, you could look at the h5 file format.

This allows you to group datasets in named containers. You can then read them from disk later on one by one i.e. you don't need to read the whole dataset into memory.

Here is an example of a function that would save such a dataset:

def save_h5(h5_filename, data, labels, descr=None,
            data_dtype='float32', label_dtype='float32'):
    """Create a compressed .h5 file containing:
    data    : numpy array
    labels  : numpy array
    descr   : text description ofthe data contained (must be a string)

    if os.path.exists(h5_filename):
        # prevent overwriting a file
        sys.exit('File already exists!')

    h5_fout = h5py.File(h5_filename)

        compression='gzip', compression_opts=4,

        compression='gzip', compression_opts=4,

    if descr is not None:
            'description', data=descr)


For the meaning of the parameters, head over the documentation.

You can write a similar function to make accessing the saved h5 file. This really is a flexible way to save data, and it can be compressed with one of the best known (widely-spread) algorithms in the open-source world: gzip! There are also other possibilities implemented.

On a side note, if you want to minimise the possibility of corruption, you could consider saving each panel/DataFrame (whichever method you go for) into separate files, and then make copies/backups.

Additionally, you said:

I tried using pickle and csv, but once a line was corrupted I lost the whole file.

... the beauty of a simple csv file is that you can actually open it in notepad or a spreadsheet and usually find the line which is "broken" and fix/delete it. Pickle, on the other hand, is a little more complicated to debug.

  • $\begingroup$ About this csv comment: my csv file for some reason became empty. The file size is still there (almost 1gb) but when opening it shows nothing. $\endgroup$ Jun 15, 2018 at 16:03
  • $\begingroup$ Opening in a 1Gb csv file ina standard viewer isn't a good idea, and some programmes will truncate it (i.e. won't show more than X lines) - Excel for example does this. Perhaps trying to do so opened the file but didn't close it properly. $\endgroup$
    – n1k31t4
    Jun 15, 2018 at 16:31
  • $\begingroup$ Not the case though. When trying to open it with pandas, it’s empty as well. I think something went wrong and somehow corrupted the file. I’ve been having problems with non utf-8 strings or something like that. $\endgroup$ Jun 15, 2018 at 17:44
  • $\begingroup$ That's weird that is it still showing 1Gb on disk. Can you measure the size with import os; os.path.getsize('path/to/file.csv')? If you're on Windows, it might be worth trying [this] (support.microsoft.com/en-us/help/17119/…) $\endgroup$
    – n1k31t4
    Jun 15, 2018 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.