I have a model that predicts the lifespan of a horse. The dataset has samples from 1980 to 2019 and among the features there is one called birth_date, labeled with the lifespan in years for each horse. The problem is knowing that a horse usually lives between 20 and 30 years, if we look from 1980 to 1990 we have the full list of horses and their lifespans, but from 2000 to 2019 we see only samples of horses that were born and died within this time span but not the ones that are currently living, therefore the birth_date is a biased feature.

Is there a way to use the birth_date feature without having to worry about the biased data, or some technique to minimize his effects on the final predictions?

  • 1
    $\begingroup$ How do you know the lifespan(target variable) of horses that were born on 2019? they are probably still alive, what is the number you have for still living horses? $\endgroup$ – yoav_aaa May 28 '19 at 11:16
  • $\begingroup$ we only have the data of the ones that have born and died already, therefore we know nothing about the currently living ones. Looking at this data someone may get the wrong idea, like horses lifespan is drastically sorter within the last years. $\endgroup$ – Peter Schwarz May 28 '19 at 11:23
  • $\begingroup$ So your entire data set is biased, not only the feature.. $\endgroup$ – yoav_aaa May 28 '19 at 11:26

What you are describing are censored observations in survival analysis. Indeed, those who are still alive will obviously die but just at a later, observable date in a time frame past our study date. By counting them as survived on a logistic regression we will clearly bias our model to favor survivals for those particular horses, unrealistically.

Hence, you have survival data and should approach it like so. The Cox proportional hazards rate model allows you to relate hazard rate functions to predictors, and while many use linear models for the "predictor" part there are other ML methods like xgboost and random forests that have the ability to return Cox PH models. These models allow for censored observations as you describe.

  • $\begingroup$ based on the OP's comment, I think the OP is dealing with truncated (not left-censored) data because he doesn't even observe the living horses... not sure if that changes anything for survival analysis but I think the usual way way of defining the likelihood doesn't apply to truncated data... (but I'm no expert on Cox ph models) $\endgroup$ – oW_ Jun 25 '19 at 19:25
  • $\begingroup$ @oW_ rereading the post and I can see how some may interpret his post like that. Indeed, if he isn't even observing horses who have survived to the end of the study that is truncation, not censoring. But then, I don't really understand how the dataset is "biased" anymore. If you simply ignore those who are censored then your observations will only be of those horses who have died. Those who die in 2000-2019 are still legitimate deaths, though below average from what is expected. $\endgroup$ – aranglol Jun 25 '19 at 23:55
  • $\begingroup$ So would a regression using a target variable = age at death not be appropriate? I don't see a problem here if you are only working with horses who are known to have died within specific time frames. $\endgroup$ – aranglol Jun 26 '19 at 0:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.