Calculate correlation between two sensors regarding the time

Question

I am new in all this, so I am sorry if I am asking something stupid. I have two-time series - the measurement of the value by two sensors every 5 min and I want to see are the measurements correlated in time (are the measurements similar). So far now I only found questions related to autocorrelation and classic correlation. My dataframe looks like this:

                  id      time                      sensor1      sensor2    
                  1       24.1.2020. 00:00:00        0.052         0.631             
                  1       24.1.2020. 00:05:00        0.812         0.102            ....
                  1       24.1.2020. 00:10:00        0.326         0.500             ....
                  1       24.1.2020. 00:15:00        1.021         0.999             ....
                  1       24.1.2020. 00:20:00        1.033         1.000             ....

and so on for 10 days.

So, time here is a really important aspect because measurements depend on part of the day, of climate condition during the day and so on. I saw on some question that people are suggesting pandas function:

corr = df['value1'].corr(df['value2'])

But as I see, this line doesn't include the time. Also if you have some course for data science to recommend, it will be appreciated. I am already taking something.

Answer 1

Your suggested method should work, assuming the two column are in the same dataframe with the same timestamps as the index. This means that they are already aligned in time and therefore the time is implicitly encoded.

If there are any missing datapoints, you could impute them (fill them) with something like the previous value, which is usually best for time-series data. Replacing with e.g. the mean of the entire time-series might be misleading because there are temporal fluctuations/cycles involved.

df.fillna(method='ffill')

You could also simply plot them to see which kind of correlation value you might expect, doing something like this:

import matplotlib.pyplot as plt    # <-- needed if running in script (not a Jupyter notebook)
df[["sensor1", "sensor2"]].plot()
plt.plot()