What are the advantages of HDF compared to alternative formats? What are the main data science tasks where HDF is really suitable and useful?
$\begingroup$
$\endgroup$
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
4
Perhaps a good way to paraphrase the question is, what are the advantages compared to alternative formats?
The main alternatives are, I think: a database, text files, or another packed/binary format.
The database options to consider are probably a columnar store or NoSQL, or for small self-contained datasets SQLite. The main advantage of the database is the ability to work with data much larger than memory, to have random or indexed access, and to add/append/modify data quickly. The main *dis*advantage is that it is much slower than HDF, for problems in which the entire dataset needs to be read in and processed. Another disadvantage is that, with the exception of embedded-style databases like SQLite, a database is a system (requiring admnistration, setup, maintenance, etc) rather than a simple self-contained data store.
The text file format options are XML/JSON/CSV. They are cross-platform/language/toolkit, and are a good archival format due to the ability to be self-describing (or obvious :). If uncompressed, they are huge (10x-100x HDF), but if compressed, they can be fairly space-efficient (compressed XML is about the same as HDF). The main disadvantage here is again speed: parsing text is much, much slower than HDF.
The other binary formats (npy/npz numpy files, blz blaze files, protocol buffers, Avro, ...) have very similar properties to HDF, except they are less widely supported (may be limited to just one platform: numpy) and may have specific other limitations. They typically do not offer a compelling advantage.
HDF is a good complement to databases, it may make sense to run a query to produce a roughly memory-sized dataset and then cache it in HDF if the same data would be used more than once. If you have a dataset which is fixed, and usually processed as a whole, storing it as a collection of appropriately sized HDF files is not a bad option. If you have a dataset which is updated often, staging some of it as HDF files periodically might still be helpful.
To summarize, HDF is a good format for data which is read (or written) typically as a whole; it is the lingua franca or common/preferred interchange format for many applications due to wide support and compatibility, decent as an archival format, and very fast.
P.S. To give this some practical context, my most recent experience comparing HDF to alternatives, a certain small (much less than memory-sized) dataset took 2 seconds to read as HDF (and most of this is probably overhead from Pandas); ~1 minute to read from JSON; and 1 hour to write to database. Certainly the database write could be sped up, but you'd better have a good DBA! This is how it works out of the box.
-
$\begingroup$ Can we say that HDF is a columnar storage format? I'm not overly proficient with either of these, but the HDF samples I've worked with do seem to present external features of column based data aggregation. $\endgroup$ Commented Apr 17, 2018 at 19:42
-
1$\begingroup$ Both databases and HDF provide "the ability to work with data much larger than memory", so I don't think it is accurate to say that this is an advantage of databases over HDF since (what they call 'partial i/o' is a core feature davis.lbl.gov/Manuals/HDF5-1.8.7/UG/12_Dataspaces.html $\endgroup$ Commented May 10, 2019 at 21:07
-
$\begingroup$ "a database is a system (requiring admnistration, setup, maintenance, etc) rather than a simple self-contained data store" - HDF is a binary file format - it requires tools to read and write too. If it was a human readable format, then maybe this argument in favor of HDF over RDMS would be stronger. You can use tools to dump a database to a single binary file too... $\endgroup$– RyanCommented Mar 9, 2020 at 13:11
-
1$\begingroup$ As of 2022, I would argue that the argument in favour of HDF5 vs other binaries is a lot weaker. Parquet is better supported outside of the python ecosystem $\endgroup$– saolofCommented Oct 28, 2022 at 13:04
$\begingroup$
$\endgroup$
1
One benefit is wide support - C, Java, Perl, Python, and R all have HDF5 bindings.
Another benefit is speed. I haven't ever seen it benchmarked, but HDF is supposed to be faster than SQL databases.
I understand that it is very good when used with both large sets of scientific data and time series data - network monitoring, usage tracking, etc.
I don't believe there is a size limitation for HDF files (although OS limits would still apply.
answered Jun 10, 2014 at 12:57
-
5$\begingroup$ From personal experience I'd add that built-in documentation/label is huge. Now all my datasets can can be stored with explicit records of where they came from, sampling frequency, anomalies, etc. etc. $\endgroup$ Commented Jun 10, 2014 at 13:04
$\begingroup$
$\endgroup$
To add, check out ASDF in particular their paper ASDF: A new data format for Astronomy; ASDF tries to improve upon HDF5 and the paper describes some downsides of HDF5 format.
$\begingroup$
$\endgroup$
HDF5 pros
- supported by
pandasout of the box - therefore, supported by
jupyter-notebookout of the box - good support by other software
- includes good amount of metadata
- single file can store multiple data frames
- each dataframe includes good amount of metadata (I think)
- thus:
- it's great for making sense of data, it's self-describing
- it's great for experimentation
HDF5 cons
- with naive tools: file must fit in RAM
- with a bit more care: the dataframe you access must fit in RAM, and
- the file must be in local storage
- each dataframe has a schema, it's homogenous
- some cloud systems don't support it out of the box (👉 aws)
- thus:
- not for humongous data
- not for network storage (e.g. s3)
- not for cases where schema changes over time
