0
$\begingroup$

I have a data table of daily values for the past 2 years that looks like this, and I need to calculate the correlations between the data in Python. I have no background in data science, so I am afraid my methods shown below were faulty. What would be the correct way to find, for example, how much does my daily productivity correlate with the tempo of my music or air humidity? And be sure the correlation is really valid (something to do with the p-value?)

id productivity total_hours swdev_hours temp pressure humidity fit_steps danceability energy loudness tempo
1 31 13.38 2.6 9.06 1011.0 53.0 0 0.716 0.759 -7.181 105.015
2 65 11.35 5.14 6.78 1012.0 65.0 0 0.908 0.669 -2.827 112.238
3 0 0.0 0.0 0.0 0.0 0.0 0 0.0 0.0 0.0 0.0
4 56 13.57 3.85 8.87 1000.0 68.0 0 0.776 0.697 -6.594 92.548
5 43 11.71 2.12 6.22 1011.0 72.0 0 0.662 0.609 -11.422 127.934
6 64 12.52 5.05 7.54 1021.0 68.0 0 0.276 0.739 -8.851 79.8
7 41 9.67 2.28 5.41 1019.0 67.0 0 0.0 0.0 0.0 0.0
8 56 12.15 2.81 5.58 1018.0 70.0 4825 0.647 0.844 -3.756 146.967
9 44 12.37 0.03 3.79 1028.0 75.0 5350 0.0 0.0 0.0 0.0
10 45 7.7 0.01 4.29 1030.0 77.0 1597 0.399 0.761 -6.318 140.084
11 34 10.19 1.23 2.22 1029.0 53.0 2171 0.399 0.761 -6.318 140.084

This is the code I used for doing a correlation on a pandas dataframe, but I am absolutely not sure if it is correct approach and need some pointers:

import pandas as pd
from io import StringIO
from scipy.stats.stats import pearsonr

csv_str = """1,31,13.38,2.6,9.06,1011.0,53.0,0,0.716,0.759,-7.181,105.015
2,65,11.35,5.14,6.78,1012.0,65.0,0,0.908,0.669,-2.827,112.238
3,0,0.0,0.0,0.0,0.0,0.0,0,0.0,0.0,0.0,0.0
4,56,13.57,3.85,8.87,1000.0,68.0,0,0.776,0.697,-6.594,92.548
5,43,11.71,2.12,6.22,1011.0,72.0,0,0.662,0.609,-11.422,127.934
6,64,12.52,5.05,7.54,1021.0,68.0,0,0.276,0.739,-8.851,79.8
7,41,9.67,2.28,5.41,1019.0,67.0,0,0.0,0.0,0.0,0.0
8,56,12.15,2.81,5.58,1018.0,70.0,4825,0.647,0.844,-3.756,146.967
9,44,12.37,0.03,3.79,1028.0,75.0,5350,0.0,0.0,0.0,0.0
10,45,7.7,0.01,4.29,1030.0,77.0,1597,0.399,0.761,-6.318,140.084
11,34,10.19,1.23,2.22,1029.0,53.0,2171,0.399,0.761,-6.318,140.084"""

csv_io = StringIO(csv_str)
col_names = ['id', 'productivity','total_hours','swdev_hours','temp','pressure','humidity','fit_steps','danceability','energy','loudness','tempo']
df = pd.read_csv(csv_io, sep=",", header=None,names=col_names)

column_name = "productivity"
res = df[df.columns].corr(method="pearson", min_periods=10)[column_name][:]
# min_periods == min occurrences to count?
print(res)
# id              0.107193
# productivity    1.000000
# total_hours     0.766020
# swdev_hours     0.714475
# temp            0.667112
# pressure        0.784574
# humidity        0.829126
# fit_steps       0.118479
# danceability    0.496831
# energy          0.516037
# loudness       -0.317449
# tempo           0.418411
# Name: productivity, dtype: float64

How can I be sure that for example the "humidity" correlation index of 0.82 is valid? That humid air can somehow cause me to be more productive, is better for my brain blood flow or something? Or is it simply correlation and has no value at all? Corr index of 0.82 seems like a really high number, but can I base my decisions on this, when I have only 2 years of data (around 720 data points). What does it really mean? How can I use the p-value to ensure validity? How should I interpret the result, isn't anything above correlation index 0.5 insanely high?

$\endgroup$

1 Answer 1

0
$\begingroup$

First of all, it is important to understand what the output of .corr() means. The value you get is the correlation between the columns. In other words, you test how strong the relationship is. However, this has nothing to do with causation. You mentioned "humid air can cause me to be more productive". This is a false statement. What you know is that based on your data, whenever there was a high humid air metric, you had also great productivity. But there is absolutely no link to the actual causation.

I would advice you to read this article to understand this concept more https://en.wikipedia.org/wiki/Correlation_does_not_imply_causation

So, to answer your question. Yes, there is a strong correlation of productivity and humidity, but we don't know if there is a causation there. You need to run an experiment to find that out.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.