3
$\begingroup$

I'm trying to come up with a data structure to predict water visibility in a lake. I have some measured samples but would like to take other features into the equation.

As an example, I would like to get weather data such as rain and temperature for the past 7 days of an event. I got this data from BigQuery which seems to work fine. But sometimes it does not find data for all the past 7 days. How would I handle such a case? What would be a good way to structure my features? I thought something like:

dayofyear,temp,water_temp,temp-1,rain-1,dayofyear-1,temp-2,rain-2,dayofyear-2,....,temp-7,rain-7,dayofyear-7,visibility

While the last one would be the field I like to predict at some point.

$\endgroup$
6
$\begingroup$

There are three main approaches to handling missing data.

  1. Impute - use some method to fill in the missing values with reasonable guesses. You could interpolate between two time points, take the average value over all time points, or use a variety of other techniques leveraging co-occurrence of other variables to get a reasonable estimate.

  2. Ignore - some methods can just ignore missing data, and not use it in the model at all

  3. Utilize - for cases where data is not missing-at-random, missingness itself can be an informative feature. You could include missing values as another data point to model your output.
$\endgroup$
  • $\begingroup$ thanks. Looks like I am going to interpolate the missing data or get another source where data is more complete. $\endgroup$ – Chris Jul 29 '17 at 9:07
3
$\begingroup$

Interpolation seems like it would make sense in this case: any time you miss a day, take an average of the before and after.

As an aside, I don't think you have to give up on the missing weather values so easily. There are a variety of R packages that simplify getting weather for an arbitrary location from someone like Weather Underground with only a couple lines of code.

$\endgroup$
  • $\begingroup$ Hi. Thanks for the note. I was actually looking at Weather Underground but the free API only allows 500 requests per day and it requires 8 request per event. Could take a while to process the old data with it. $\endgroup$ – Chris Jul 29 '17 at 9:05
  • $\begingroup$ If you have to do a lot, this one is a little tougher to use but queries NOAA direct. They likely have a higher limit. cran.r-project.org/web/packages/rnoaa/index.html $\endgroup$ – CalZ Jul 30 '17 at 14:36
  • $\begingroup$ thanks for the hint. What does this differ from NOAA GSOD from BigQuery? $\endgroup$ – Chris Jul 30 '17 at 19:54
  • $\begingroup$ NOAA is a USA government agency. Once you get a token, I see no mention of a request limit. $\endgroup$ – CalZ Jul 31 '17 at 11:48
  • $\begingroup$ Indeed but I guess BigQuery GSOD tables are in sync with the data provided by NOAA cloud.google.com/bigquery/public-data/noaa-gsod $\endgroup$ – Chris Jul 31 '17 at 13:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.