Best file format for transfer of EHR data

Question

I am working on a clinical trial where we have several sites sending us EHR data. The sites are currently sending the data in excel files. I have a feeling someone's opening the files because 3 of the files have 64,999 rows exactly, and excel 2007 cuts off at 65,000.

I am working in python, but I am trying to prevent the people at the local sites from opening the files in excel.

What's the best format for the files to be sent to me such that they can't be opened and cut off in excel? csv unfortunately can also be opened in excel.

Answer 1

If your are using python, I suggest using Numpy's npy format. An NPY file is a NumPy array file created by the Python software package with the NumPy library installed. It contains an array saved in the NumPy (NPY) file format. NPY files store all the information required to reconstruct an array on any computer, which includes dtype and shape information.

You can also consider using hdf or hdf5 formats both of which are supported in Python. Hierarchical Data Format (HDF) is an open source file format for storing huge amounts of numerical data. It’s typically used in research applications (meteorology, astronomy, genomics etc.) to distribute and access very large datasets without using a database. One can use HDF5 data format for pretty fast serialization to large datasets.