From my understanding in order to improve accuracy with ensemble models you need a wide range of independent ensemble methods. I was wondering whether using the ouput of a random forest model as one of the inputs for a neural network where the other input is the original data and the targets remain the same could improve the model? Why add extra complexity? basically the problem is a multi-dimensional regression problem and although the random forest gets a smaller MSE the neural network is bette at preservering some of the properties of the target labels. Therefore, I was wondering if by putting these two models together I would get a lower MSE while preserving some of the properties. Is it worth a shot? or will it just drastically overfit?
2 Answers
I suggest you test it by cross-validation to prevent overfitting. However, I guess because of a strong relationship between this variable and output your work will not be better. Instead, try to use an ensemble.
I read something about deep learning which in that way they use several models as a sequence and use results of previous models as inputs of the next models. Maybe if you continue this work it will be in some way a deep learning.
I'm working on prediction of time series and my stacked LSTMs alone does not capture well the trend and seasonality of my data. I decided to add features from a Holts Winter regression as an additional input and the result is way better. It slightly overfits but in my case that's not an issue.
In my case, I like the fact that my statistic (holts winter) model can capture well the global trend, and the network can find other non linear relations between datapoints. I guess it's a good idea to combine several models.
