Does RandomForest convergence imply I can solve a problem with a NN too?

I'm trying to perform a regression on a dataset, and I've been testing a few models, mostly for practice.

I was able to get good results with a RandomForestRegression model, as you can see in the scatter plot:

So, I tried to solve the same problem with a NN (again, I'm doing it mostly to practice), but the results are definitely bad. As you can see in the top part of the following picture, the loss on the training and test set (red and green lines, respectively) seem to somewhat converge, but their value is still higher than the one obtained with the RF (the horizontal blue line). Also, the scatter plots are horrible, since the model is basically always predicting the mean of the data.

The model is a simple MLP, with one hidden layer. I'm using a tanh activation function for the hidden neurons. Since the losses seem to have converged, I don't know if using a smaller learning rate or training the model for more epochs would improve the results.

However, my question is more general, independently of this specific result. I have the general understanding that NNs are a more flexible than a RandomForest, so I am wondering if the fact that I obtained a good result with a RandomForest, automatically implies that there should exist some NN configuration which could provide a result at least as good as the one obtained with the RF.
Is this true, and I just have to fix the NN model until I find the right configuration? Or is it possible that this problem can be properly modelled by a RF, but not by a NN?

• For NNs (MLPs) the universal approximation theorem holds which means using enough layers and training can approximate any from a class of functions. For random forests the theorem is a variant of that of ensemble methods. Nevertheless one (configurated) model may have better performance than a different (configurated) model. Do not generalise it Feb 1 '21 at 19:53
• As @NikosM. already mentioned, NN are univerisal function approximators and should be capable to do a simple linear regression. I assume you have a mistake in your code. Feb 4 '21 at 11:33
• The problem might be in the hyperparameters of you NN, try using linear activation instead Feb 4 '21 at 23:06

To me those are separate things since both models have a different cost function to be optimized.

On the other hand you could combine those models by constructing embeddings based on random forest splits and then using those embeddings as inputs for a neural network.

Toy example shows that there is a non-trivial configuration of a neural net that can get as good results as those obtained a by a random forest:

from sklearn.datasets import load_iris
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import RandomTreesEmbedding

X, y = load_iris(return_X_y = True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 42, test_size = .2)

params = dict(n_estimators=100,
max_depth=None,
min_samples_split=2,
min_samples_leaf=1,
min_weight_fraction_leaf=0.0,
max_leaf_nodes=None,
min_impurity_decrease=0.0,
min_impurity_split=None,
random_state=42,
verbose=0)

mlp = Pipeline([("embeddings", RandomTreesEmbedding(**params)),
("model", MLPClassifier(activation = "identity",hidden_layer_sizes=(1000,), max_iter = 10000, random_state = 42))]).fit(X_train, y_train)

rf = Pipeline([("model", RandomForestClassifier(**params))]).fit(X_train, y_train)

mlp.score(X_test, y_test)
rf.score(X_test, y_test)