I'm still a newbie in Data Science and I found the Titanic question and solution here in Kaggle. I've been trying to understand the solution yet I still can't grasp why he used those percentiles for the numerical feature distribution part.

Here's the analysis:

  • Total samples are 891 or 40% of the actual number of passengers on board the Titanic (2,224).
  • Survived is a categorical feature with 0 or 1 values.
  • Around 38% samples survived representative of the actual survival rate at 32%.
  • Most passengers (> 75%) did not travel with parents or children.
  • Nearly 30% of the passengers had siblings and/or spouse aboard.
  • Fares varied significantly with few passengers (<1%) paying as high as $512.
  • Few elderly passengers (<1%) within age range 65-80.

Here's the code:

train_df.describe()
# Review survived rate using `percentiles=[.61, .62]` knowing our problem description mentions 38% survival rate.
# Review Parch distribution using `percentiles=[.75, .8]`
# SibSp distribution `[.68, .69]`
# Age and Fare `[.1, .2, .3, .4, .5, .6, .7, .8, .9, .99]`

These questions were asked a couple of times in the comment section of the solution yet I can't find any concrete answer. Can anyone try to explain to me in great detail the reason behind the usage of the specific percentiles?

  • 1
    Here the author tried to find some biggest differences in the variables. Biggest meaning it could be important for the prediction. You can start with some high level percentiles (0,25,50,75,100) and then improve resolution where you see something interesting. When you find the point of the change of the data then you only need two points to prove your idea. Which you see here. But it does not mean that the author started his analysis this way. – keiv.fly Dec 5 at 22:25

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.