I am trying to do some analysis on some data that comes from special glasses that track a few things including pupil size and gaze velocity. I would like to calculate the correlation between two glasses on two different people. At the moment I cannot use
df.correlate() because the timestamps are not identical and therefore the data looks something like this:
index | ts | r_person | l_person ----------------------- 0 | 23 | 3.0 | NAN 1 | 25 | NAN | 3.2 2 | 28 | 3.1 | NAN 3 | 32 | 3.0 | NAN
I was wondering if there was still any way to calculate a correlation directly.
At the moment I was thinking of possible filling the NAN values with the averages of the data points above and below. For example row 2 column
r_person would become $3.05$.
This would be less trivial than it seems because even at the start it wasn't always one data point R one data point L and after cleaning the data it has become less so. In other words multiple NAN values might appear in the same column as you can see in the example. I can still deal with that by just spreading out the average. My second technique was going to try and merge the values together if they were close enough. Bearing in mind the data was collected at 50hz.
My question is whether anyone has a quicker or better way of aligning the data without losing it or changing it too much?