Sorry for the vagueness of the title; I'll explain what I mean. I'm doing the Kaggle Titanic beginner tutorial. The label you're interested in is the "survived" rate (0 or 1), which you want to predict in the test data.

One of the first things I did was plot the average survival rate for each of the features, for each one, taking the average survival rate for that value of the feature. For example, for the "Sex" feature, you can look at the average survival rate for males and females, which gives you good info you can use.

For features like "Fare" and "Age", which are more continuous, it wouldn't really make sense to take the average in the same way, because there aren't necessarily many people with the same exact age. So, I did "banding", where I do a histogram-like thing of saying that everyone from 0-10 is one band, 10-20 another band, etc, and those bands are now categorical features that can be averaged over well (this is a good move, right? at least initially?).

This is what I get:

enter image description here

That's the background. Now, here's my question. I want to start with a simple model like linear regression. For some features, like Sex, or Pclass, it's clear that a linear model could fit it very well. For some however, like AgeBand, there's not really a clear, monotonic trend for increasing AgeBand...but there definitely is a difference in the survival rate between different bands!

So what you could do is, sort those "non monotonic" features by the survival rate, so it is monotonic, and then use them for linear regression. Is that a good idea or no?

For example, here's the AgeBand feature sorted by the survival rate:

enter image description here


I would not recommend doing this in general as it will likely lead to overfitting.

While a particular reordering might improve fit on your dataset, the goal is to create a model with good predictive power. If you received never before seen data from an independent second, third, one hundredth Titanic, would the ordering still be the same? I doubt it. You'd probably be incorrectly using the somewhat arbitrary ordering fixed on the first, real Titanic on new data where the ordering is no longer the same.

I think the story goes that the young and the elderly got preferential treatment and this is why their survival rate was higher. Therefore your AgeBand curve should be approximately U-shaped. (Which it is, approximately.) If you insist on linear regression, I don't see how you could ever (directly) include this nonlinear (U-shaped) effect.

The fewer assumptions (particularities) go into the choice of features, the better. It would then make more sense to me to reorder such that you pick bands alternately from the two ends (starting from the middle): 3, 4, 2, 5, 1, 6, 0, 7 (or in reverse order, starting from the ends). Or, even better, pool the age bands into coarser groups: {1,2,3}, {4,5,6,7}, {0}. I think these are justified by your exploratory data analysis.

When I look at the range of values in the AgeBand plot ($0.33$ to $0.55$), this variable does not actually explain as much as Pclass, Sex, FareBand. In the end you might end up dropping it from your variables anyway, so once again, the more attention you pay to it, the more you're at risk of overfitting.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.