I am currently building my first machine learning model using the titanic dataset. After the data exploration, I decided to focus my attention on the 'Ticket' feature. One thing I have noticed about this feature is that it is not unique per each passenger; this had led me to believe that other features can be extracted from this variable:
- Is_Group -> to indicate if the ticket represents a group booking
- Group_Size -> the number of passengers in each group.
# save the tickets that appear more than once (i.e. group tickets)
_ = (data.Ticket.value_counts()>1).to_dict()
ls = []
for key in _:
if _[key]==True:
ls.append(key)
#extract the feature
data['Is_Group'] = data['Ticket'].apply(lambda x: 1 if x in(ls) else 0)
# create another dict containing the number of counts per each ticket
group_size = (data.Ticket.value_counts()).to_dict()
# extract the feature from the mapping
data['Group_Size'] = data['Ticket'].map(group_size).fillna(0)
The reason I am doing this is because I wanted to explore the nature of the relationship between the extracted features from Ticket and the target Survived (and later decide how to deal with outliers in the SibSp and Parch feature):
data.groupby(['Is_Group','Group_Size'])['Survived'].describe()
From the above table I can see that group size 2/3 have almost a 60% and 70% chance of survival. Now, this led me to think that there is a correlation (or at least some sort of relationship between group size and Survived). Therefore, I decided to create a correlation matrix to make sure that is the case.
As I expected there is a correlation between Is_Group and Group_Size (as they have been extracted from the same feature) but there is no correlation between these extracted features and Survived. Hence my, confusion. I thought given the high mean values of Survived for Group_Size (2,3) there was a relationship but clearly I am getting something wrong here.
Can anyone help clear this doubt of mine?