1
$\begingroup$

I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of it.

Some options appear to be:

  • HDFS
  • HDF5
  • HDFS3
  • PyArrow
$\endgroup$
  • $\begingroup$ When I want to got 5 mts in distance, I would rather walk than to take a car. $\endgroup$ – Kiritee Gak Mar 26 at 10:23
  • $\begingroup$ I think HDF5 is very good for you, your data size is small, I am working on h5 files it's fast. $\endgroup$ – Hunar A.Ahmed Mar 26 at 10:32
  • 1
    $\begingroup$ Just leave it as CSV you don't need to do anything $\endgroup$ – arhwerhwe Mar 26 at 11:27
  • 1
    $\begingroup$ Why not dump the dataframe to_pickle ? Easy, low memory, compression supported and fast loading without specifying columns or other parameters ... $\endgroup$ – n1tk Mar 26 at 18:24
4
$\begingroup$

With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.

Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.

$\endgroup$
  • 1
    $\begingroup$ +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it. $\endgroup$ – Bilkokuya Mar 26 at 14:21
  • $\begingroup$ That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well... $\endgroup$ – B Seven Mar 26 at 17:05
3
$\begingroup$

You can find a nice benchmark for every approach in here.

enter image description here

$\endgroup$
1
$\begingroup$

Your data size is not that much huge, but there are some debates whenever you deal with big data What is the best way to store data in Python and Optimized I/O operations in Python. They all depend on the way the serialisation occurs and the policies which are taken in different layers. For instance, security, valid transactions and such things. I guess the latter link can help you dealing with large data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.