1
$\begingroup$

I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of it.

Some options appear to be:

  • HDFS
  • HDF5
  • HDFS3
  • PyArrow
Improve this question
$\endgroup$
4
  • $\begingroup$ When I want to got 5 mts in distance, I would rather walk than to take a car. $\endgroup$
    – Kiritee Gak
    Mar 26, 2019 at 10:23
  • $\begingroup$ I think HDF5 is very good for you, your data size is small, I am working on h5 files it's fast. $\endgroup$
    – Hunar
    Mar 26, 2019 at 10:32
  • 1
    $\begingroup$ Just leave it as CSV you don't need to do anything $\endgroup$
    – arhwerhwe
    Mar 26, 2019 at 11:27
  • 1
    $\begingroup$ Why not dump the dataframe to_pickle ? Easy, low memory, compression supported and fast loading without specifying columns or other parameters ... $\endgroup$
    – n1tk
    Mar 26, 2019 at 18:24

3 Answers 3

Reset to default
5
$\begingroup$

With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.

Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it. $\endgroup$
    – Bilkokuya
    Mar 26, 2019 at 14:21
  • $\begingroup$ That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well... $\endgroup$
    – B Seven
    Mar 26, 2019 at 17:05
3
$\begingroup$

You can find a nice benchmark for every approach in here.

enter image description here

Improve this answer
$\endgroup$
1
$\begingroup$

Your data size is not that much huge, but there are some debates whenever you deal with big data What is the best way to store data in Python and Optimized I/O operations in Python. They all depend on the way the serialisation occurs and the policies which are taken in different layers. For instance, security, valid transactions and such things. I guess the latter link can help you dealing with large data.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.