Imputing time data for an event that hasn't occurred yet

Question

Suppose you are trying to predict if lightning will strike in a given location in a given month. (Please ignore the meteorology here, this is not my actual problem, just a hypothetical instance of the general problem.)

So for each location, every day you compute some features like latitude, longitude, air pressure, etc. One feature that might be of interest is the number of days that have passed without a lightning strike.

Now suppose you have a location where lightning has not struck yet, but you still want to include it in your training data. The logically correct value for 'days_since_last_strike' is NULL. But your learning algorithm cannot handle NULL values.

What is a sensible value to replace NULL with?

I thought a negative number might work, but should it be -1, -9999, -INT_MAX? Is there a better solution?

Answer 1

As so often, the answer is: "It depends". In this case, it depends on the algorithm / model / method you are going to use.

There are some methods, e.g. tree-based methods, that can handle NULL-values. In this case, it could make sense not to impute at all.
Note at this point the difference between informative missing (in our case: there was no lightning before) and non-informative missing (in our case: I don't know if there was a lightning). Some methods only work well with non-informative missings. In this case, I would create a dummy variable (see below).

If you use some method based on linear elements (e.g. linear or logistic regression, neural networks), I would suggest the following approach: Create two variables: one contains your variable (or 0 imputed for NULL value) and the other one is a binary dummy variable with is 1, if there is a NULL value, and 0 otherwise.

Why to do so?
Look at the linear term: $\beta_1x_1+\beta_2x_2+...$. If $x_1$ is the value and $x_2$ is the flag, then the we have two cases:

1. If the value is not NULL, then $\beta_1x_1+\beta_2x_2=\beta_1x_1$
2. If your value is NULL, then $\beta_1x_1+\beta_2x_2=\beta_2$

So your model will learn the best value for imputation.

Answer 2

It's very easy to over-complicate things. Agree with Broele that it depends.

Let's look at what data you actually have:

1. Since t0 (when you started collecting data), you know this location has not had a strike.
2. You have no evidence as to what happened before t0.

-> Generally, put the total number of days you have captured, the model will work out the rest without needing dummy variables. And it will be quite robust to whatever model you choose (especially when normalising your features).

• If this doesn't work, then proceed down the increased complexities of imputation.
• Do not fill it with 0 days, this will confuse the heck out most simple models, and will bias you to higher complexity models.