Can HDF5 be reliably written to and read from simultaneously by separate python processes?

Question

I'm writing a script to record live data over time into a single HDF5 file which includes my whole dataset for this project. I'm working with Python 3.6 and decided to create a command line tool using click to gather the data.

My concern is what will happen if the data gathering script is writing to the HDF5 file and the yet-to-be ML application tries to read data from the same file?

I took a look at The HDF Group's documentation about HDF5 parallel I/O, but that didn't really clear things up for me.

Answer 1

HDF5 parallel I/O will not solve this problem. That technology is primarily intended for performance, not for collision avoidance.

What you want is know as SWMR (single-writer/multiple-reader):

Data acquisition and computer modeling systems often need to analyze and visualize data while it is being written. It is not unusual, for example, for an application to produce results in the middle of a run that suggest some basic parameters be changed, sensors be adjusted, or the run be scrapped entirely.

To enable users to check on such systems, we have been developing a concurrent read/write file access pattern we call SWMR (pronounced swimmer). SWMR is short for single-writer/multiple-reader. SWMR functionality allows a writer process to add data to a file while multiple reader processes read from the file.

SWMR was first included in HDF5 version 1.10.0 released on 2016-03-30

Concurrent Access to HDF5 Files - Single Writer/ Multple Reader (SWMR)

The Single Writer/ Multiple Reader or SWMR feature enables users to read data concurrently while writing it. Communications between the processes and file locking are not required. The processes can run on the same or on different platforms as long as they share a common file system that is POSIX compliant.