I am trying to predict a continuous 'Y' variable using a variety of algorithms and feature engineering techniques. My issue is that Y is extremely variable and I reached a asymptote in accuracy.
This is the structure of my feature variables (with swapped variable names)
|Screw Width (mm)
|Screw Height (mm)
|Screw Angle (degrees)
|Car Model Id
|Car Age (Years)
My target variables go as followed:
|Speed without Screw
|Speed with Screw
Which I merge into a single variable Y:
The range of the Delta can go from -2000. to 165,000. Please note I swapped the variable names again so I'm not in reality predicting speed.
Currently my R2 score is 0.9 and the mean prediction error rate is 25%. I want to get it down to 15%
What I have tried so far:
- Constraining the Input and Sensitivity Analysis on the variables for current Regressor Choice
For some time I tried:
- Splitting the data (e.g. run on Screw Width < 0.5 mm)
- Build a model on the split data
- Record the new R2 score/Mean Error
To my surprise it wasn't one variable or range of that variable that was causing huge variance as repeating this experiment gave me similar results. For example filtering heavily on the variables and repeating steps 1-3 gave me results like as followed:
- Using different regressors to predict a single continuous variable
I tried using a simple Linear Regressor, MLP Regressor, Gradient Boosting and Random Forest. I haven't played too much with the parameters of each regressor and am not sure if that is my downfall. Currently a RandomForestRegressor is giving me best results but each regressor converges around a similar range.
- Simple Feature Engineering:
I tried OHE for non numerical variables, Log Transforming the Data. The difference was noticeable but still small.
- Running more data
I have close to 250,000 simulations (or rows) and reached a bit of an asymptote at 25% accuracy after 150,000 simulations with the strategies/algorithms I deployed.
Summary: I need advise on how to tackle this problem. Do I just keep on randomly trying methods/manipulating the data until something sticks. Or is there a better path moving forward to get a better accuracy?
What have I not tried?