How to find correlations between data over time?

Question

I have daily data of my sales and a daily update of how the COVID-19 cases increase. My daily sales contains information on my customer and on my product. My end goal would be to see if certain products are correlated (sold more or less) after covid than before. Or if certain regions had more sales before or after COVID.

My initial thought would be to do some kind of event analysis and compare if I have more sales before and after that certain event. Does anyone has some experience with this or can link some papers/references on this topic?

Answer 1

You can read about time varying correlation, you can obtain the correlation at each point in time based on some fixed number of last days, enough days to make it smooth but not many so as to be able to see changes in correlation. You can also use exponential smoothers to give more weight to recent observations. Also the dynamic conditional correlation might be interesting to you.

Answer 2

First, you will have to aggregate the sales data at day level and product level for a particular region or overall.

Aggregation of Sales at day and product level

e.g.

day    product  sales

day_1  p1       sales_1
day_2  p1       sales_2
day_3  p1       sales_3
day_4  p1       sales_4
...
day_n  p1       sales_n

similarly, for each product. e.g. p2, p3 ....

COVID-19 cases at day level

Then, You might be having the figures of COVID-19 for that region or ooverall.

day   covid_19_count

day_1  count_1
day_2  count_2
day_3  count_3
day_4  count_4
...
day_n  count_n

Merge the data for the sales and COVID-19 cases

Then, create a single table having day, sales and covid_19_count for that region.

day    product  sales   covid_19_count

day_1  p1       sales_1 count_1
day_2  p1       sales_2 count_2
day_3  p1       sales_3 count_3
day_4  p1       sales_4 count_4
...
day_n  p1       sales_n count_n

Calculation of correlation

Then, calculate the correlation between sales and covid_19_count for that product for that particular region or overall.

Answer 3

Since you are dealing with correlation between two time series in this example, you do not want to simply rely on standard correlation measures, as these will not tell you anything about whether there is significance behind that correlation.

For this problem, I would be inclined to use cointegration to examine if there is a theoretically justified correlation between sales and COVID-19 cases.

Let's take an example using Python.

Two random arrays of numbers with mean = 100 and standard deviation = 10 are generated in Python.

mu_1, sigma_1 = 100, 10 # mean and standard deviation
mu_2, sigma_2 = 100, 10 # mean and standard deviation
data1 = np.random.normal(mu_1, sigma_1, 1000)
data2 = np.random.normal(mu_2, sigma_2, 1000)

Cointegration results are generated in Python using ts.coint:

>>> coin_result = ts.coint(data1, data2)
>>> coin_result

(-31.93013915989978, 0.0, array([-3.90743646, -3.34225305, -3.04869817]))

With a p-value of 0.0, this implies that the two series are perfectly cointegrated, which we would expect since they both follow a normal distribution and have the same mean and standard deviation. If the p-value were greater than 0.05, then the series is not indicated to be cointegrated at the 5% level of significance.

However, reality will be different, as two series never follow exactly the same distribution. Even if two series are cointegrated, it will never be perfect cointegration.

In your case, you could see if sales fluctuations and COVID-19 cases show cointegration. However, I would caveat that you should be using active cases instead of cases outright.

This is because the total number of cases can never decrease, it can only increase or flatline. Given that you will inevitably see fluctuations in sales, then the vastly differing distributions will mean that your series will never be indicated as cointegrated - which may not be true in reality.

In terms of looking at sales in isolation, you could also use a Chow Test to examine for the presence of a structural break, i.e. if the series is significantly different before and after the outbreak.

To summarize, I would be inclined to:

1. Test for cointegration between active cases and sales

2. Of those segments that do show cointegration between active cases and sales:

a. Run a Chow test to see if there is a structural break between sales pre and post COVID-19

b. Compute descriptive statistics for sales across these two periods, e.g. average sales pre and post, standard deviation of sales pre and post, etc.

This way will likely give you better insights as opposed to simply relying on correlation measures, which are likely to be misleading for the reasons I have mentioned.