0
$\begingroup$

My current dataset has a shape of 5300 rows by 160 columns with a numeric target variable range=[641, 3001].
distribution of y_true
That’s no big dataset, but should in general be enough for decent regression quality. The columns are features from different consecutive process steps.

The project goal is to predict the numerical variable, with the satisfactory object to be very precise in the area up too 1200, which are 115 rows (2,1%). For target variables above 1200 the precision can be lower than in the area [640, 1200]. The target-variable is normally distributed with its mean ~1780 (25%: 1620, 75%: 1950) and variance of 267.5.

prediction vs actual:
prediction vs actual
residual plot:
residual-plot
My problem is (see plots above), that no matter what I try, the range of predictions (y_hat) is very limited and rather random (Training RMSE ~300, Test RMSE ~450), best test-mean-abs-error for y-values <= 1200 ~= 120.

I’ve already tried:

  • feature cleaning
  • process step wise addition of features to compare model performance/information gain
  • feature generation
  • derive new features (by business logic)
  • generate features
    • cross-product of features
    • differences to previous rows
    • differences between features
    • differences per feature to mean
    • durations based on timestamps
  • normalizing, scaling
  • log-transformation of target variable
  • Over- &/ Under-Sampling
  • various algorithms (using GridSearchCV for hyper-parameter tuning):
  • sklearn [SVR, RandomForrestRegressor, LinearRegression, Lasso, ElasticNet]
  • xgboost
  • (mxnet.gluon.Dense)

What would be your approach? Do you have any advice what technique I could try or what I've probably missed? Or if it's more likely that the training data simply doesn't fit well on the target variable?

Improve this question
$\endgroup$
3
  • $\begingroup$ There's something wrong. Almost all your y_hat is < 1500 (see your plot). But more than three quarters of your y_true are >1500. So how can the residuals be centred at 0? It should be that the majority of your predictions are smaller than true values. Aren't your plots inconsistent? $\endgroup$
    – f.g.
    Oct 14, 2018 at 9:44
  • $\begingroup$ For the model training I've used different loss functions. The loss function that resulted in the plot penailzes higher values, trying to be better in the lower value area. $\endgroup$
    – Michael_S
    Nov 4, 2018 at 15:27
  • $\begingroup$ Regardless... The plots are inconsistent. I don't think they can both be correct. If you y_hat and y_true relationship is as depicted in plot 1, then plot 2 shouldn't be possible. $\endgroup$
    – f.g.
    Nov 5, 2018 at 16:22

1 Answer 1

Reset to default
0
$\begingroup$

Your residuals are huge, which is not surprising, given that your data is very variable, a linear model may not be the best choice for this task. You could try transforming your data (log, sqrt) depending on the nature of your data to reduce the variability, but as I said, your variability is huge.

Alternatively you could try modeling the variance with a mixed model if it makes sense for your data, given some additional knowledge of some variable.

Other then that you could try a different algorithm for this task.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I don't think you read his post. He tried your suggestions already. $\endgroup$
    – f.g.
    Oct 14, 2018 at 9:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.