1
$\begingroup$

I am running a regression on the following data set to predict white wine quality

Data set link: https://archive.ics.uci.edu/ml/datasets/wine+quality

Data csv name: winequality-white.csv

Features: 1 - fixed acidity,2 - volatile acidity,3 - citric acid,4 - residual sugar,5 - chlorides,6 - free sulfur dioxide,7 - total sulfur dioxide,8 - density,9 - pH,10 - sulphates,11 - alcohol

Target variable: quality

test split: 33%


Model One:

random forest regression scikit-learn implementation

Pre-processing: sklearn standard scaler(though not required for RFR)

Hyper params: 'rf_regr__max_features': 'auto', 'rf_regr__max_leaf_nodes': None, 'rf_regr__min_samples_leaf': 1, 'rf_regr__min_samples_split': 5, 'rf_regr__n_estimators': 10

Test R2 score: 0.84


Model Two

Fully connected neural network

Pre-processing: sklearn standard scaler

Hyper params: optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),loss=tf.keras.losses.mse, callback = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=10)
chpt = tf.keras.callbacks.ModelCheckpoint('wine_model',monitor='val_loss',save_best_only=True)

Toplogy:

tf.keras.Input(shape=(num_features)),

tf.keras.layers.Dense(16,activation='relu'),

tf.keras.layers.Dense(8,activation='relu'),

tf.keras.layers.Dense(4,activation='relu'),

tf.keras.layers.Dense(2,activation='relu'),

tf.keras.layers.Dense(2,activation='relu'),

tf.keras.layers.Dense(1,activation='linear')               

I have experimented with various other topologies and learning rate/other HP.

Best R2 score achieved on test data: 0.33.


Why is there such a missive difference in test R2 score between Random forest(0.84) and Neural regression(0.33)?

Also, I observed that the neural net is not even able to fit the training data even with 6-7 hidden layers. Test score is starting to decline after adding more than 4 layers.

$\endgroup$

1 Answer 1

1
$\begingroup$

Neuronal networks requires a lot of data to be trained. Moreover, if you have small quantity of data, it often suffers of overfitting as you commented. Generally, neuronal networks are used in more complex problems. Another possible cause is that neuronal networks works better with scaled features, have you done it? Finally, NN has a lot of parameters to tune: layers, neurons, activation functions,.. finding the right is complicated; also, it is an art.

As you can see, depending on your dataset, some algorithms fits much better than others, it's an art.

$\endgroup$
2
  • $\begingroup$ Thanks for your comment, yes the data was scaled with standard scaler. And experiments were done with various number of layers and with all Hyper params except Activation function which was fixed to relu. I guess the issue is related to the small size of the data set. However, is there any good mathematical reason as to why NN require so much of data compared to random forest? Yes-the number of trainable params is a lot more but why do we require so many trainable params to fit the training data in the first place?And why is it not the case with random forest? $\endgroup$ Apr 18, 2022 at 8:46
  • $\begingroup$ NN use backpropagation to find the best algorithm; it is powerful, but it need a lot of data and iteration to learn. Whereas RF is just a combination of decision tree (which are easy to create). They are just complete different types of algorithms. $\endgroup$ Apr 19, 2022 at 7:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.