Regression - random forest vs Fully connected neural net

Question

I am running a regression on the following data set to predict white wine quality

Data set link: https://archive.ics.uci.edu/ml/datasets/wine+quality

Data csv name: winequality-white.csv

Features: 1 - fixed acidity,2 - volatile acidity,3 - citric acid,4 - residual sugar,5 - chlorides,6 - free sulfur dioxide,7 - total sulfur dioxide,8 - density,9 - pH,10 - sulphates,11 - alcohol

Target variable: quality

test split: 33%

---

Model One:

random forest regression scikit-learn implementation

Pre-processing: sklearn standard scaler(though not required for RFR)

Hyper params: 'rf_regr__max_features': 'auto', 'rf_regr__max_leaf_nodes': None, 'rf_regr__min_samples_leaf': 1, 'rf_regr__min_samples_split': 5, 'rf_regr__n_estimators': 10

Test R2 score: 0.84

---

Model Two

Fully connected neural network

Pre-processing: sklearn standard scaler

Hyper params: optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),loss=tf.keras.losses.mse, callback = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=10)
chpt = tf.keras.callbacks.ModelCheckpoint('wine_model',monitor='val_loss',save_best_only=True)

Toplogy:

tf.keras.Input(shape=(num_features)),

tf.keras.layers.Dense(16,activation='relu'),

tf.keras.layers.Dense(8,activation='relu'),

tf.keras.layers.Dense(4,activation='relu'),

tf.keras.layers.Dense(2,activation='relu'),

tf.keras.layers.Dense(2,activation='relu'),

tf.keras.layers.Dense(1,activation='linear')               

I have experimented with various other topologies and learning rate/other HP.

Best R2 score achieved on test data: 0.33.

---

Why is there such a missive difference in test R2 score between Random forest(0.84) and Neural regression(0.33)?

Also, I observed that the neural net is not even able to fit the training data even with 6-7 hidden layers. Test score is starting to decline after adding more than 4 layers.

Answer 1

Neuronal networks requires a lot of data to be trained. Moreover, if you have small quantity of data, it often suffers of overfitting as you commented. Generally, neuronal networks are used in more complex problems. Another possible cause is that neuronal networks works better with scaled features, have you done it? Finally, NN has a lot of parameters to tune: layers, neurons, activation functions,.. finding the right is complicated; also, it is an art.

As you can see, depending on your dataset, some algorithms fits much better than others, it's an art.