Feature Engineering - correlation with binary outcome - Titanic Dataset - Ticket feature

Question

I am currently building my first machine learning model using the titanic dataset. After the data exploration, I decided to focus my attention on the 'Ticket' feature. One thing I have noticed about this feature is that it is not unique per each passenger; this had led me to believe that other features can be extracted from this variable:

• Is_Group -> to indicate if the ticket represents a group booking
• Group_Size -> the number of passengers in each group.

# save the tickets that appear more than once (i.e. group tickets)
_ = (data.Ticket.value_counts()>1).to_dict()
ls = []
for key in _:
    if _[key]==True:
        ls.append(key)
#extract the feature
data['Is_Group'] = data['Ticket'].apply(lambda x: 1 if x in(ls) else 0)

# create another dict containing the number of counts per each ticket
group_size = (data.Ticket.value_counts()).to_dict()
# extract the feature from the mapping
data['Group_Size'] = data['Ticket'].map(group_size).fillna(0)

The reason I am doing this is because I wanted to explore the nature of the relationship between the extracted features from Ticket and the target Survived (and later decide how to deal with outliers in the SibSp and Parch feature):

data.groupby(['Is_Group','Group_Size'])['Survived'].describe()

From the above table I can see that group size 2/3 have almost a 60% and 70% chance of survival. Now, this led me to think that there is a correlation (or at least some sort of relationship between group size and Survived). Therefore, I decided to create a correlation matrix to make sure that is the case.

As I expected there is a correlation between Is_Group and Group_Size (as they have been extracted from the same feature) but there is no correlation between these extracted features and Survived. Hence my, confusion. I thought given the high mean values of Survived for Group_Size (2,3) there was a relationship but clearly I am getting something wrong here.

Can anyone help clear this doubt of mine?

Answer 1

The thing is that you are looking at correlation with binary outcome. There as some counter intuitive things when doing that (see for example: https://stats.stackexchange.com/questions/103801/is-it-meaningful-to-calculate-pearson-or-spearman-correlation-between-two-boolea).

Outside than that, the features do not appears so much predictive.

More specifically for Is_group: Individual tickets (Is_group = 0), yields à 30% mean, while individuals with group tickets have around 50%. This is not really predictive.

Similarly, the average probability doesn't evolve monotonically of group size (from 0.3, to 0.6 then 0.7, then droping to 0.5 followed by 0 twice then back to 0.23). It doesn't seems really appropriate to use linear correlation to measure the relation between group_size and output here.