I'm writing a script to record live data over time into a single HDF5 file which includes my whole dataset for this project. I'm working with Python 3.6 and decided to create a command line tool using click to gather the data.

My concern is what will happen if the data gathering script is writing to the HDF5 file and the yet-to-be ML application tries to read data from the same file?

I took a look at The HDF Group's documentation about HDF5 parallel I/O, but that didn't really clear things up for me.


HDF5 parallel I/O will not solve this problem. That technology is primarily intended for performance, not for collision avoidance.

What you want is know as SWMR (single-writer/multiple-reader):

Data acquisition and computer modeling systems often need to analyze and visualize data while it is being written. It is not unusual, for example, for an application to produce results in the middle of a run that suggest some basic parameters be changed, sensors be adjusted, or the run be scrapped entirely.

To enable users to check on such systems, we have been developing a concurrent read/write file access pattern we call SWMR (pronounced swimmer). SWMR is short for single-writer/multiple-reader. SWMR functionality allows a writer process to add data to a file while multiple reader processes read from the file.

SWMR was first included in HDF5 version 1.10.0 released on 2016-03-30

Concurrent Access to HDF5 Files - Single Writer/ Multple Reader (SWMR)

The Single Writer/ Multiple Reader or SWMR feature enables users to read data concurrently while writing it. Communications between the processes and file locking are not required. The processes can run on the same or on different platforms as long as they share a common file system that is POSIX compliant.

  • $\begingroup$ We need to write HDF5 from Spark; will the Parallel I/O work for that? $\endgroup$ – vy32 Mar 15 at 1:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.