# How to handle missing data for machine learning

I'm trying to come up with a data structure to predict water visibility in a lake. I have some measured samples but would like to take other features into the equation.

As an example, I would like to get weather data such as rain and temperature for the past 7 days of an event. I got this data from BigQuery which seems to work fine. But sometimes it does not find data for all the past 7 days. How would I handle such a case? What would be a good way to structure my features? I thought something like:

dayofyear,temp,water_temp,temp-1,rain-1,dayofyear-1,temp-2,rain-2,dayofyear-2,....,temp-7,rain-7,dayofyear-7,visibility


While the last one would be the field I like to predict at some point.

There are three main approaches to handling missing data.

1. Impute - use some method to fill in the missing values with reasonable guesses. You could interpolate between two time points, take the average value over all time points, or use a variety of other techniques leveraging co-occurrence of other variables to get a reasonable estimate.

2. Ignore - some methods can just ignore missing data, and not use it in the model at all

3. Utilize - for cases where data is not missing-at-random, missingness itself can be an informative feature. You could include missing values as another data point to model your output.
• thanks. Looks like I am going to interpolate the missing data or get another source where data is more complete. Jul 29 '17 at 9:07

Interpolation seems like it would make sense in this case: any time you miss a day, take an average of the before and after.

As an aside, I don't think you have to give up on the missing weather values so easily. There are a variety of R packages that simplify getting weather for an arbitrary location from someone like Weather Underground with only a couple lines of code.

• Hi. Thanks for the note. I was actually looking at Weather Underground but the free API only allows 500 requests per day and it requires 8 request per event. Could take a while to process the old data with it. Jul 29 '17 at 9:05
• If you have to do a lot, this one is a little tougher to use but queries NOAA direct. They likely have a higher limit. cran.r-project.org/web/packages/rnoaa/index.html
– CalZ
Jul 30 '17 at 14:36
• thanks for the hint. What does this differ from NOAA GSOD from BigQuery? Jul 30 '17 at 19:54
• NOAA is a USA government agency. Once you get a token, I see no mention of a request limit.
– CalZ
Jul 31 '17 at 11:48
• Indeed but I guess BigQuery GSOD tables are in sync with the data provided by NOAA cloud.google.com/bigquery/public-data/noaa-gsod Jul 31 '17 at 13:36