I'm trying to perform a regression on a dataset, and I've been testing a few models, mostly for practice.
I was able to get good results with a RandomForestRegression model, as you can see in the scatter plot:
So, I tried to solve the same problem with a NN (again, I'm doing it mostly to practice), but the results are definitely bad. As you can see in the top part of the following picture, the loss on the training and test set (red and green lines, respectively) seem to somewhat converge, but their value is still higher than the one obtained with the RF (the horizontal blue line). Also, the scatter plots are horrible, since the model is basically always predicting the mean of the data.
The model is a simple MLP, with one hidden layer. I'm using a tanh activation function for the hidden neurons. Since the losses seem to have converged, I don't know if using a smaller learning rate or training the model for more epochs would improve the results.
However, my question is more general, independently of this specific result.
I have the general understanding that NNs are a more flexible than a RandomForest, so I am wondering if the fact that I obtained a good result with a RandomForest, automatically implies that there should exist some NN configuration which could provide a result at least as good as the one obtained with the RF.
Is this true, and I just have to fix the NN model until I find the right configuration? Or is it possible that this problem can be properly modelled by a RF, but not by a NN?