4
$\begingroup$

As part of my thesis I've done some experiments that have resulted in a reasonable amount of time-series data (motion-capture + eye movements). I have a way of storing and organizing all of this data, but it's made me wonder whether there are best practices out there for this sort of task.

I'll describe what I've got, and maybe that will help provide some recommendations. So, I have an experiment that requires subjects to use their vision and move their body to complete a task. Each task is one trial, and each subject performs multiple trials to complete the experiment. During a trial I record the movement and the eye tracker (~200 channels) at regularly sampled time points (~100Hz). I store these in a CSV file (one file per trial), with one row per time point, and one column per variable (e.g., left-fingertip-x, left-fingertip-y, left-fingertip-z, etc. for the mocap, and left-eye-x, left-eye-y for the eyes).

Associated with each trial is some metadata such as the experimental condition of the trial (e.g., how fast a target in the trial is moving, say). I store these values in the CSV filename itself, using a "key=value" sort of syntax.

While this works well enough for my purposes, it's really ad-hoc! I'd like to get a sense of whether other people have solved problems like this, and, if so, how?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
8
$\begingroup$

There are two solutions that are worth looking at:

InfluxDB is an open source database platform specifically designed for time series data. The platform includes many optimized functions related to time and you can collect data on any interval and compute rollups/aggregations when reporting. The company recently launched a query app called Chronograf. I have not used this - but if its no good, you can also check out Grafana (which is very widely used and stable).

The alternative strategy you may want to pursue is an elasticsearch index. Elasticsearch is great at running aggregations and other mathematical functions on data. Many use it to store server log data and then query said data using Kibana.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Neat, I hadn't heard of InfluxDB. It looks like it's designed for web analytics but could probably handle the data I'm collecting -- I sample at ~100Hz on ~200 channels. Thanks for the pointer! $\endgroup$
    – lmjohns3
    Aug 21, 2015 at 14:24
  • 1
    $\begingroup$ It's actually designed for IoT sensor data :-) $\endgroup$
    – Tom Griffin
    Aug 21, 2015 at 15:57
0
$\begingroup$

As part of ELK, you will find a pretty nice time series analysis open source call timelion. Once you catch how it works, it is an excellent framework.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.