2
$\begingroup$

I got different time series. Each time series contains different attributes, which have data entries at different points of time (timestamps). Each time series is labeled and has a label attribute.

I made a graphic showing the structure of the data.

enter image description here

It is important to me to have a structure which is persistable as a csv. I tried this with nested dataframes, but the nested dataframe is not saved correctly.

The dataset is about 250 Megabytes, stored in a numpy array of dimension (3, 1200, 2501, 10).

I am looking for an efficient approach to create such a dataframe with the given data structure, such that it is persistable with pandas and file size still efficient.

Edit:

The file format does not necessarily need to be csv.

$\endgroup$
8
  • 1
    $\begingroup$ [In my exp] Well Pandas or CSV isn't made for hierarchical data storage. If we still want to store it that way, a simple thing would be to keep single single measure files on to the disk and rebuild the df whenever needed? Atleast I would try this as my 101. Also did you try grouping by the measure_k column and then dumping that as JSON? $\endgroup$ – Aditya Apr 7 at 14:52
  • 1
    $\begingroup$ Are you fine if we repeat that measure_k col equal to Time_n? Then we can cache it easily at some extra space overhead. Pandas would be happy then. Interesting question! $\endgroup$ – Aditya Apr 7 at 14:56
  • 1
    $\begingroup$ Can you try something like this? stackoverflow.com/questions/17349574/… ; Basically you just specify the Index cols back if you don't want to go for the hacky solution. NB Be careful if you have null values. $\endgroup$ – Aditya Apr 7 at 15:06
  • 1
    $\begingroup$ Just checking, What did you go with then? Thanks! $\endgroup$ – Aditya Apr 14 at 12:41
  • 1
    $\begingroup$ I went with what you suggested, it works. But I also went with pickling the dataframe, which preserves quite everything I need :) $\endgroup$ – Thomas Christopher Davies Apr 15 at 9:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.