Similar output from random forest and neural network

Question

I used a training data set to train both a random forest and a neural network (one hidden layer). Then I compared how both systems perform on a test data set. Interestingly, both turned out to have about the same prediction probabilities. The forest classified 86% of the data correctly into category 0 or 1, the network achieved 85%. As there is 75% of the data in category 0, I am not too happy with the result and hoped for something better.

I then analyzed, which data sets were classified incorrectly. It turned out that there is a large overlap: 86% percent of the data, which was classified incorrectly by the forest, was also classified wrongly by the network. I then compared the probabilities which were attributed to those test samples by the forest and the network. Also the probabilities were almost always comparable so that when the forest was sure that sample A belongs to category 0, then also the network was sure that 0 is the right category. Also, if the probabilities of the forest were around 50% for both categories, also the network was "unsure" which category was correct.

Is such an outcome somehow obvious? Is there an explanation?

Answer 1

I think that your neural network is probably learning the same features as your Random Forest. I do not think it is obvious because it is hard to say what the neural network is learning, but in this case based on the your model it might be working similar.