Need help! How do you test for a correlation between two data sets?

Question

I'm doing a research project and want to test for correlation between different data sets.

For example, I want to test if there is a correlation between median house prices and homeless population in the US by year. Here is some made up data for the problem:

Year 2000 , House price $260,000, homeless pop 330,000

2005 - 270,000 - 315,000

2010 - 285,000 - 320,000

2015 - 330,000 - 340,000

2020 - 400,000 - 370,000

I want to then get (r) to measure the correlation between these two data sets and compare that strength of correlation to other data sets (for example, median house price and rates of domestic violence in the US)

Thank you for the help!

Answer 1

It would depend on your specific problem statement, if you do not want to consider this as a time series data i.e. do not want to take year into account you would simply consider the correlation values between the home price and homeless population versus home price and domestic violence; whichever value will be high in magnitude (positively or negatively correlated) will be strongly correlated than the other

data_df['Home Price'].corr(data_df['Homeless pop'])

data_df['Home Price'].corr(data_df['Domestic Violence rate'])

If you want to consider time factor ; then you would have to convert the date column into datetime column and then consider three different time series

1. Year and home price
2. Year and homeless pop
3. Year and domestic violence rate

And then you can use granger causality test for causality or cross correlation to see the correlation between time series. You can refer to this post as well -

https://towardsdatascience.com/computing-cross-correlation-between-geophysical-time-series-488642be7bf0#:~:text=Cross%2Dcorrelation%20is%20an%20established,inference%20on%20the%20seismic%20data.