How to handle missing data for machine learning

Question

I'm trying to come up with a data structure to predict water visibility in a lake. I have some measured samples but would like to take other features into the equation.

As an example, I would like to get weather data such as rain and temperature for the past 7 days of an event. I got this data from BigQuery which seems to work fine. But sometimes it does not find data for all the past 7 days. How would I handle such a case? What would be a good way to structure my features? I thought something like:

dayofyear,temp,water_temp,temp-1,rain-1,dayofyear-1,temp-2,rain-2,dayofyear-2,....,temp-7,rain-7,dayofyear-7,visibility

While the last one would be the field I like to predict at some point.

Answer 1

There are three main approaches to handling missing data.

1. Impute - use some method to fill in the missing values with reasonable guesses. You could interpolate between two time points, take the average value over all time points, or use a variety of other techniques leveraging co-occurrence of other variables to get a reasonable estimate.

2. Ignore - some methods can just ignore missing data, and not use it in the model at all

3. Utilize - for cases where data is not missing-at-random, missingness itself can be an informative feature. You could include missing values as another data point to model your output.

Answer 2

Interpolation seems like it would make sense in this case: any time you miss a day, take an average of the before and after.

As an aside, I don't think you have to give up on the missing weather values so easily. There are a variety of R packages that simplify getting weather for an arbitrary location from someone like Weather Underground with only a couple lines of code.