Survival regression with major event that won't happen

Question

I would like to do some survival regression about the duration before the "death" of an individual. The final purpose is to know, given an individual, how long it should take before he'll most likely "die" (probability of the survival function to be less than 0.1 for instance).

My problem here is that I have, in my training set, a variable that influences a lot my target variable, but is not available for the test set (and won't happen in real life).

Let's say my training data is the following:

id   status   poison_time  death_time     sex
 0     1          90           92          f
 1     0          90          150          f 
 2     1          90           91          f  
 3     1          60          130          m
 4     0          60          150          m
 5     1          60           62          m

With :

• status = 1 for a dead person and 0 for a censored data
• poison_time : time corresponding to the injection of a poison
• death_time : time of the death or last follow-up
• sex : sex of the individual (not relevant here, imagine a bunch of useful variables)

I can't just ignore the influence of poison_time: although for some individuals, the poison won't be as effective (individual with id 3, or individuals that ended up right-censored). It has a real impact on death_time.

In my test data the poison is not injected, but I still would like to have a good idea of "how long should it take before an individual most likely die", knowing my other variables (sex, etc.)

Is it possible to still have relevant results with such corrupted data as a training set?

Answer 1

If I understand your problem correctly, I think it's possible¹, but you have to do some extra work and you may be limited with what models you can use.

First, you have a time-varying dataset, so that must be handled correctly. Your poison comes long after birth, and that's important to model. Otherwise you are biasing your model. Ex: Suppose everyone is poisoned at age 90, if they live that long. Then just the act of being poisoned is a signal of living a long time, when in reality, it will hasten the death of the subject. Give this a read too, for many more arguments why it's important to model time-varying covariates.

Okay, so I took your dataset, and I modelled it into a time-varying dataset (see below). Notice:

1. A subject could have more than 1 rows (in your case, everyone will have at most two rows, one for prior to poisoning, one for after)
2. Each row for a subject is mutually exclusive (see start and stop columns). The status is True iff they died at the end of that interval.
3. poisoned is a boolean flag for activating after they were poisoned.

    start     poisoned sex   stop  id  status
0       0            0   f     90   0   False
1      90            1   f     92   0    True
2       0            0   f     90   1   False
3      90            1   f    150   1   False
4       0            0   f     90   2   False
5      90            1   f     91   2    True
6       0            0   m     60   3   False
7      60            1   m    130   3    True
8       0            0   m     60   4   False
9      60            1   m    150   4   False
10      0            0   m     60   5   False
11     60            1   m     62   5    True

The above matrix has the exact same information, but presented differently.

---

Okay, so we now need to pick a model that can handle time-varying datasets. Let's take a step back and ask "after fitting a model on the training data, can I "transport" that to the testing data (which has no poisonings)?" I argue that, yes, this is valid, and will be more valid you have individuals who died before being poisoned. What your model will do is understand the risk factors associated with your variables given poisonings / no poisoning. Hence, when you move to your test data, the risk factors are already learned and prediction can be done. Here's an analogy: I'm studying the student exam scores on Canadian and American students, and have other variables like lunch, grade, background etc and possibly interactions between them. The objective of "fitting" is to learn the relationship of lunch on exam scores given the other variables. So when I focus my model on only Canadian students in the test data, my model is still valid.

(However, if I had no Canadian students in my training data, then the model's validity can be questioned if I wish to focus on Canadian students in the test data. This is why I hope there are subjects in your training data who died before being poisoned).

---

We can fit the time-varying dataset using Cox's proportional hazard model. The (linear) model's output looks like:

---
             coef exp(coef)  se(coef)     z    p  -log2(p)  lower 0.95  upper 0.95
poisoned    17.60  4.41e+07   2670.86  0.01 0.99      0.01    -5217.18     5252.38
sex         -0.82      0.44      1.25 -0.66 0.51      0.97       -3.28        1.64
---

From your description, it sounds like your test data is not time varying. That's fine, it still fits into the time-varying framework: simply each test subject has a single row. You can choose to leave the poisoned variable out, or just leave it as 0 throughout. The latter is probably easier, especially if you have interactions.

¹ I could be missing some important gotcha that renders this all useless. Generally, working with time-varying data is easy to mess up causality/data leakage.