Smoothing target variable

Question

I am training a regression model (using quantile regression forest) to forecast crop yield deviations from trend (residuals) using weather variables with different lag times. Trying to improve the accuracy and confidence of my results, I recently tested replacing the target variable with a smoothed version of it (computed with locally weighted scatterplot smoothing, LOWESS) using as independent variable a feature with no lag time (i.e., with lag time = 0) trying to remove noise from the measured data. In the figure below, the crosses represent the observed values, the red dots are outliers, and the gray line is the smoothed version of the dependent variable.

I got significant improvements in the result, but something doesn't seem right with this approach.

I have been looking into the use of smoothing techniques in machine learning and have found that, indeed, smoothing is a technique used in data preprocessing, feature engineering, and data mining for noise filtering (e.g., here or here; or here, applied for time series forecasting). On the one hand, it sounds logical to remove noise from the target variable to estimate "true values" derived from the process I am trying to model; however, I've learned that the preprocessing is applied to features (explanatory variables) and I am not sure that smoothing the target variable is a valid procedure. To summarize:

1. Is it valid to replace the target variable with a smoothed version of it?

2. If it is, given that the independent variable used to smooth the target variable is not available a priori, how should I proceed?

   a. train the model with the smoothed target variable and test it with the raw target variable; or

   b. train and test the model with the smoothed target variable.

Any thoughts will be appreciated.

Answer 1

Is it valid to replace the target variable with a smoothed version of it?

It does not sound like a valid approach.

I’m not convinced that you removed the noise and not legitimate effects. If you consider any two neighbor points on your graph, the difference in target variable is likely to be driven by a number of factors (not shown on you plot). It might appears as noise (big jumps) because we don’t account for other factors.

Imagine that your target variable is the height of a student and you smooth using the height ~ age loess, because you observe some big jumps in height e.g. between 17 and 17.5 y.o. The problem is that half of your students are from Netherland (the tallest nation in Europe). If you smooth your target as above you’ll never be able to get good fit (to the original target, of course) even if you include the nationality as a feature.

In other words, applying such smoothing you disregard the influence of all other factors and implicitly assume that the target can be modelled as a smooth function of a single (in your case unobservable / unavailable) factor.

In general, removing the noise might help, but I guess it’s very difficult to differentiate between true effects and the noise in practice. Instead of pre-processing the target I would just control for overfitting (e.g. with some regularization or early stopping) and let the model smooth the data.