1
$\begingroup$

I am currently working through Kaggle's titanic competition and I'm trying to figure out the correlation between the Survived column and other columns. I am using numpy.corrcoef() to matrix the correlation between the columns and here is what I have:

The correlation between pClass & Survived is: [[ 1.         -0.33848104]
 [-0.33848104  1.        ]]

The correlation between Sex & Survived is: [[ 1.         -0.54335138]
 [-0.54335138  1.        ]]

The correlation between Age & Survived is:[[ 1.         -0.07065723]
 [-0.07065723  1.        ]]

The correlation between Fare & Survived is: [[1.         0.25730652]
 [0.25730652 1.        ]]

The correlation between Parent-Children & Survived is: [[1.         0.08162941]
 [0.08162941 1.        ]]

The correlation between Sibling-Spouse & Survived is: [[ 1.        -0.0353225]
 [-0.0353225  1.       ]]

The correlation between Embarked & Survived is: [[ 1.         -0.16767531]
 [-0.16767531  1.        ]]

There should be higher correlation between Survived and [pClass, sex, Sibling-Spouse] and yet the values are really low. I'm new to this so I understand that a simple method is not the best way to find correlations but at the moment, this doesn't add up.

This is my full code (without the printf() calls):

import pandas as pd
import numpy as np

train = pd.read_csv("https://raw.githubusercontent.com/oo92/Titanic-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Titanic-Kaggle/master/test.csv")

survived = train['Survived']
pClass = train['Pclass']
sex = train['Sex'].replace(['female', 'male'], [0, 1])
age = train['Age'].fillna(round(float(np.mean(train['Age'].dropna()))))
fare = train['Fare']
parch = train['Parch']
sibSp = train['SibSp']
embarked = train['Embarked'].replace(['C', 'Q', 'S'], [1, 2, 3])
$\endgroup$
  • $\begingroup$ why do you think the values should be higher? $\endgroup$ – nairboon May 14 '19 at 9:56
  • $\begingroup$ Because there is a strong correlation between sex, class and survival. Women and rich passengers were most likely to survive. $\endgroup$ – Andros Adrianopolos May 14 '19 at 9:59
4
$\begingroup$

On a side note, I don't think correlation is the correct measure of relation for you to be using, since Survived is technically a binary categorical variable.

"Correlation" measures used should depend on the type of variables being investigated:

  1. continuous variable v continuous variable: use "traditional" correlation - e.g. Spearman's rank correlation or Pearson's linear correlation.
  2. continuous variable v categorical variable: use an ANOVA F-test / difference of means
  3. categorical variable v categorical variable: use Chi-square / Cramer's V
|improve this answer|||||
$\endgroup$
  • 1
    $\begingroup$ Here is a closely related old post. $\endgroup$ – Esmailian May 18 '19 at 15:29
  • $\begingroup$ @bradS When you say ANOVA F-test/difference of means, do you mean dividing ANOVA F-test by difference of means? $\endgroup$ – Andros Adrianopolos May 19 '19 at 17:50
  • $\begingroup$ @AtillaAdrianopolos, no I mean "/" as "or". Using item 3 above as an example, use Chi-square test of independence or Cramer's V. $\endgroup$ – bradS May 20 '19 at 8:09
1
$\begingroup$

You probably encoded Women as 0 and men as 1 that's why you get a negative correlation of -0.54, because Survived is 0 for No and 1 for Yes. Your calculation actually show what you've expected. The negative correlation is only about the direction depending on your encoding, the relationship between Women and Survived is 0.54.

Similarly pClass is correlated negatively with -0.33 because the highest class (1st class) is encoded as 1 and the lowest as 3, thus the direction is negative.

You could make the relations more intuitive if you make new columns for men and women where you put 0 and 1 depending on the sex, then the correlations will have the intuitive direction (sign). The same holds for pClass.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ I've added my code. $\endgroup$ – Andros Adrianopolos May 14 '19 at 10:14
  • $\begingroup$ What if I encode male/female with 3/4 instead? They're still binary values and just might solve the problem you're raisng. $\endgroup$ – Andros Adrianopolos May 14 '19 at 10:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.