1
$\begingroup$

I'm predicting ozone concentration based on meteorological variables and ozone value of the previous day. I applied savitzky golay filter to get rid of noise in the time-series dataset.

My question is, if I want to perform feature selection, do I do it before or after applying the filter? What is the logical order? Because the feature importance is different before and after applying the filter.

Using XGBOOST, this is the feature importance before the filter: enter image description here And this is after the filter: enter image description here

I'd really be grateful for any help or information.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

I find your question confusing (this might be my fault). Let's see if I understand you correctly.

Feature selection is not applied before or after the filter. You choose to include a feature, or not. There are many ways to make a selection of features. If you use decision trees (or algorithms based on decision trees like XGBoost or random forest), then you can calculate the feature importance and use that to select features.

Your set-up to predict ozone concentration contains feature processing (the filter) and then an algorithm (boosted trees) that makes a prediction. You use a set of features, run them through a filter, you train the trees. Now you can use the filter + the trained trees to get your prediction (ozone concentration). You can also look at the feature importance of all the features that you used, make a selection, and retrain with a reduced set of features.

What makes no sense to me is to use a different set-up without the filter, to train the trees on the raw unfiltered features, and then look at the feature importance, because you have trained your trees on input data that the trees will never see in the set-up you're actually trying to use.

So I think the answer to your question is "after". :-)

Improve this answer
$\endgroup$
6
  • $\begingroup$ I thought of using the data without the filter because the feature importance after using the filter showed a very high importance of only O3_1 while the others were relatively very small, so I wasn't sure, because in the researches I've read other meteorological variables have higher importance. $\endgroup$
    – M. Grimm
    Aug 5, 2019 at 19:47
  • $\begingroup$ O3_1 is the O3 concentration from the day before or something? I don’t think it should matter but are you normalizing your input features? (Center on a mean of 0 and multiply to get a std deviation of 1) $\endgroup$
    – Paul
    Aug 6, 2019 at 23:37
  • $\begingroup$ Another point: From your graph it appears like you are using a feature like "day of year" directly, which is a problem, because the number 1 and 365 may be far apart, but the days 1 and 365 in a year are next to each year. Same for season, month, day of month, day of week. See: stats.stackexchange.com/questions/126230/… $\endgroup$
    – Paul
    Aug 7, 2019 at 9:01
  • $\begingroup$ Hmm yes, I read some articles on that point, but I also read many researches that used the features without changing/encoding them. I actually tried converting the month to its sin and cosine components, but it didn't affect the output results, nor the feature importance. That's why I reverted back to using the month only. Do you advise me to change them even if the prediction results don't get better? $\endgroup$
    – M. Grimm
    Aug 7, 2019 at 13:34
  • $\begingroup$ Decision trees can handle categorical features, but (a) you need to tell the algorithm which ones are categorical and (b) if you do that, you lose the relations between the values. If you treat day of week as categorical, each day is a category and the algorithm won’t know that Monday is close to Tuesday but far from Friday. So if there is a sensible encoding, that may give better results. $\endgroup$
    – Paul
    Aug 7, 2019 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.