Different Correlation Coefficents with different Time Ranges

Question

I built a Time-Series that displays the price of the Electricty Price in South Italy and two of their most important commodities (commodities, gas) used to produce the eletrical energy. So I ordered all these data into DataFrame where there are the following data in details:

1. First Column - Daily Price of Petroil Future during N Day;
2. Second Column - Daily Price of Gas Future during N Day;
3. Third Column - Daily Price of Dau-Ahead Eletricity Market in Italy;

The data are taken from 2010 to 2022 time range, so 12 years of historical time data. The DataFrame head looks like this:

     PETROIL       GAS  ELECTRICITY
0  64.138395  2.496172    68.608696
1  65.196161  2.482612   113.739130
2  64.982403  2.505938   112.086957
3  64.272606  2.500000   110.043478
4  65.993436  2.521739    95.260870

So on this DataFrame I tried to build the Correlation Matrix throught the Pandas metod .corr() (using the Pearson method) and faced one big issue:

If I take all 12 years as data I get:

1. almost Zero as correlation between Electricity and Petroil price;
2. low correlation (0.12) between Electricity and Gas price;

If I try to split in three time range (2010-2014; 2014-2018; 2018-2022) I get for each interval, really high correlation for both pair (electricity-gas, electricity-petroil) in a range around 0.60 to 0.90.

So I am here asking these two questions:

1. Why I get this so high difference when I split the time ranges?
2. Considering I am doing this kind of analysis to use Petroil and Gas prices to predict the electricity price, which of these two analysis should I consider? The first one (with low correlation) that considers the entire time range or the second one (with higher correlation) that is split into different time ranges?

Thank you for your answers.

Answer 1

This might be explained by the nature of your data in different market circumstances in the different timeframes, to elaborate, these are each market circumstances with a high correlation between prices of commodity A and B:

• A declines <-> B increases (strong negative correlation)
• A increases <-> B decreases (strong negative correlation)
• A increases <-> B increases (strong positive correlation)
• A decreases <-> B decreases (strong positive correlation)
• A steady <-> B steady (strong positive correlation)

In other timeframes, prices between commodity A and B can decouple. The correlation is thus 0:

• A declining has no effect on B
• B declining has no effect on A
• A increasing has no effect on B
• B increasing has no effect on A
• sometimes A increases when B decreases and sometimes the other way around (correlation is still 0 because values single out each other)

Now, if you mix those values all into one bucket, the result can not be easily interpreted, because you compare values between different market situations. This is also known as "comparing apples with oranges", where apple and orange are different market circumstances that 2 commodities trade in.

In fact, even grouping by year could then be considered too large of a timeframe to look at at once. One approach would be to figure out the different timeframes where commodity A and B are expected to be correlated (or not) as a list of boundaries between market situations (e.g. one boundary for each event driving either of the commodities' prices thus changing the correlation of the combo) and then subsequently calculating a correlation for each of these individual timeframes to make visible how events change the correlation between a commodity combo.