BACKGROUND
I have a RandomForestRegressor from scikit-learn which, for each example row, takes in four float features and predicts a float target. All numbers are scaled and lie within "reasonable" ranges. To be specific, the target float is simply the next value of one of the features. I.e., it's a time-series prediction on one of the features (which I'll refer to as the "driving" feature below).
PROBLEM
Most of the time, the model trains and serves correctly, attaining satisfactory mean-squared errors. However, every once in a while, and with the exact same data, the model learns that the target value is simply the previous value of the driving feature. I.e., all the model learns to do is "memorize" the previous value of the driving feature---so no actual fitting per se is performed.
QUESTION
Why does this happen? The code, data, and execution environment are held constant. It's really the model that behaves this way intermittently. I surmised that initialization (cf. random_state) may have something to do with this, but I feel that I don't want to introduce any kind of bias by arbitrarily setting a seed. The problem isn't so much reproducibility, but rather the breakdown of the learning into a simple memorization model.
