Validation Loss is not decreasing - Regression model

Question

I'm trying to train a regression model with 6 input features. No. of tuples - 7287. Train set - 5465 Test set - 1822

I've tried changing no. of hidden layers and hidden neurons, early stopping, shuffling the data, changing learning and decay rates and my inputs are standardized (Python Standard Scaler). Validation loss doesn't decrease.

NN_model = Sequential()


NN_model.add(Dense(128, kernel_initializer='normal',input_dim = x_train.shape[1], activation='relu'))


NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))

NN_model.add(Dense(1, kernel_initializer='normal',activation='linear'))
#adam=optimizers.Adam(lr=0.01,decay=0.0001)

NN_model.compile(loss='mse', optimizer='adam', metrics ['mean_absolute_error','mse'])
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=200)
NN_model.summary()
history=NN_model.fit(x_train,y_train,epochs=1000,batch_size=64,validation_data=[x_test,y_test],callbacks=[es])

Loss graph looks like:

Mean square error is very high and r2 score is 0.5276 for the train set and 0.3383 for the test set.

I've tried other machine learning models like Gradient Boosting Regressor, Random forest regressor, decision tree regressor but they all have high mean square error.

I'm new to keras and deep learning. How do I solve the issue?

Is this amount of training data enough for the neural network? I can't get more data.

Answer 1

1. the network architecture above is a very strange choice. When you have only 6 input features, it is weird to have so much Dense layers stacked.

2. if network is overfitting, WHERE IS DROPOUT? Why not trying some regularizers, if the latter does not help?

3. +1 for David Waterworth - correlation/causal analysis is not everything yet. Does linear regression provide better R-square values?

4. what is output(target) variable range? Maybe it should be mapped/scaled to something reasonable? (I judge from loss values).

5. activation function and initializers are important too. Try using different values, rather than relu/linear and 'normal' initializer.

EDIT: yes, this should be enough data, if your data has only 6 inputs. However, you can try augmenting data too, if it makes sense and you can make reasonable assumptions in your case - sometimes it gives difference in the long run, even if in the beginning you think it does not work.

EDIT2: with specific datasets, neural network can get into local plateau (not minima however), where it does not escape. To test this hypothesis, you can set learning rate to small value and all initializers to generate small values too - then network may not go to this plateau suddenly, but goes to global minima instead.

EDIT3: increasing batch size leads to faster but poorer convergence on certain datasets. On a smaller network, batch size = 1 sometimes makes wonders.