2
$\begingroup$

I am training a regression model (using quantile regression forest) to forecast crop yield deviations from trend (residuals) using weather variables with different lag times. Trying to improve the accuracy and confidence of my results, I recently tested replacing the target variable with a smoothed version of it (computed with locally weighted scatterplot smoothing, LOWESS) using as independent variable a feature with no lag time (i.e., with lag time = 0) trying to remove noise from the measured data. In the figure below, the crosses represent the observed values, the red dots are outliers, and the gray line is the smoothed version of the dependent variable.

enter image description here

I got significant improvements in the result, but something doesn't seem right with this approach.

I have been looking into the use of smoothing techniques in machine learning and have found that, indeed, smoothing is a technique used in data preprocessing, feature engineering, and data mining for noise filtering (e.g., here or here; or here, applied for time series forecasting). On the one hand, it sounds logical to remove noise from the target variable to estimate "true values" derived from the process I am trying to model; however, I've learned that the preprocessing is applied to features (explanatory variables) and I am not sure that smoothing the target variable is a valid procedure. To summarize:

  1. Is it valid to replace the target variable with a smoothed version of it?

  2. If it is, given that the independent variable used to smooth the target variable is not available a priori, how should I proceed?

    a. train the model with the smoothed target variable and test it with the raw target variable; or

    b. train and test the model with the smoothed target variable.

Any thoughts will be appreciated.

$\endgroup$
3
  • $\begingroup$ I'm following here $\endgroup$
    – CutePoison
    Dec 9 '21 at 9:58
  • 1
    $\begingroup$ When you say that the smoothing improved the result, did you test with the smoothed target variable or the original target? Because the first option would be an error from the point of view of evaluation. $\endgroup$
    – Erwan
    Dec 9 '21 at 23:46
  • $\begingroup$ That's right, @Erwan. I evaluated with the smoothed version of the target variable in the test set. And that's what made this approach seem wrong to me: evaluating with values that were not measured/observed but derived from a transformation. Thank you very much for your comment. $\endgroup$ Dec 10 '21 at 7:09
2
$\begingroup$

Is it valid to replace the target variable with a smoothed version of it?

It does not sound like a valid approach.

I’m not convinced that you removed the noise and not legitimate effects. If you consider any two neighbor points on your graph, the difference in target variable is likely to be driven by a number of factors (not shown on you plot). It might appears as noise (big jumps) because we don’t account for other factors.

Imagine that your target variable is the height of a student and you smooth using the height ~ age loess, because you observe some big jumps in height e.g. between 17 and 17.5 y.o. The problem is that half of your students are from Netherland (the tallest nation in Europe). If you smooth your target as above you’ll never be able to get good fit (to the original target, of course) even if you include the nationality as a feature.

In other words, applying such smoothing you disregard the influence of all other factors and implicitly assume that the target can be modelled as a smooth function of a single (in your case unobservable / unavailable) factor.

In general, removing the noise might help, but I guess it’s very difficult to differentiate between true effects and the noise in practice. Instead of pre-processing the target I would just control for overfitting (e.g. with some regularization or early stopping) and let the model smooth the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.