Format for time series data with non-trivial sampled data

Question

I have a data stream that I would like to share with some data scientists.

It is a regularly captured time series with some fields that are simple scalars, booleans. Each sample has a UTC time and fractional seconds since start of capture.

Also captured is a 3d spline that varies in size. These splines are also regularly sampled.

Additionally, there are some other multi dimensional fields as well ([x,y,z,pitch,yaw,roll]).

For simple data sets I would normally use CSV. However due to the nature of the more complex data I need a more appropriate format.

What are my options for formats that would allow easy loading into Matlab or other common data science tools?

Answer 1

Based on this article about Data Formats in Data Science, the short answer to your question is: use JSON.

This may seem natural next step after csv and no more, but it is actually quite interesting:

JSON is widely supported in many programming languages, it is a valid MIME type in Internet standards (application/json, data stream mentioned in your question) and it has several standard and non standard extensions (geoJSON, binary json). If there is not a standard structure, JSON makes it easy to invent your own, because it is easily modifiable.

According to the article JSON works as value in HADOOP. For HADOOP it is suitable, because basically JSON is plain text when you think it as a data item.

Also article mentions JSON more efficient than CSV.