I am using Python to do weather forecasting.

Here is the original data.


The input has 180 features and the meanings are:

Suppose we are doing forecast for hour k. We will use the historic weather data (past 3 hours) and the weather forecast (hour k).

For the wind generator, the 180 input features (4x11x4+4) are:

Station 1 Hour k-3: [temperature, humidity, wind speed, wind direction] ... Hour k: [temperature, humidity, wind speed, wind direction]


Station 11 Hour k-3: [temperature, humidity, wind speed, wind direction] ... Hour k: [temperature, humidity, wind speed, wind direction]

Time related features [sin(hour), cos(hour), sin(day_of_year), cos(day_of_year)]

4114 +4 = 180

And one output is the power.

The hint is, This data is not clean, in that there is some portion of bad data in the target/output data (not the input data). However, it is hard to know which data is bad. You may consider how to clean the data first (more about sample selection, not feature selection). After data cleaning, you may then consider feature selection to reduce the input dimension. There a lot of feature selection methods available in literature.

I wonder how to clean the data and then do feature selection. Thanks!


Data cleaning typically requires knowledge of the application domain.

  • Try to identify ranges of reasonable values for each of the input features, based on your understanding of what the feature represents. For instance, a wind speed of 2000 mph is probably not real. A humidity value of 300% or -50% is probably not right.

  • Also, try to work out a plausible range for how much the feature value should or should not vary from one sample to the next.

  • More generally, if you have a model for how these values might evolve, you can look for samples that are grotesquely implausible according to the model.

Visualizing the data is also typically helpful. For instance, you can look for outliers and examine them more closely with manual inspection.

  • $\begingroup$ Thank you very much! I am new to it, and it is just a course project which does not require as much experience. Your answer is really helpful! But only the output data is bad data and I don't know how to figure out. And I also want to ask how to deal with feature engineering because there are 4 features in a group. And there are 11 stations, each station records 4 hours data. Thanks! $\endgroup$ Apr 20 '17 at 0:15
  • $\begingroup$ @AgraynelChen, You're welcome. You should ask only one question per question, so feature engineering belongs in a separate question (but you should research it first; there's lots written on it, and asking a question that is already well covered in standard resources probably won't be well received). I don't know what is meant by "only the output data is bad data". $\endgroup$
    – D.W.
    Apr 20 '17 at 16:04
  • $\begingroup$ the input are correct. But many output are bad. Only wind speed and wind direction highly influence the output. $\endgroup$ Apr 21 '17 at 23:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.