# Validation Loss is not decreasing - Regression model

I'm trying to train a regression model with 6 input features. No. of tuples - 7287. Train set - 5465 Test set - 1822

I've tried changing no. of hidden layers and hidden neurons, early stopping, shuffling the data, changing learning and decay rates and my inputs are standardized (Python Standard Scaler). Validation loss doesn't decrease.

NN_model = Sequential()

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=200)
NN_model.summary()
history=NN_model.fit(x_train,y_train,epochs=1000,batch_size=64,validation_data=[x_test,y_test],callbacks=[es])


Loss graph looks like:

Mean square error is very high and r2 score is 0.5276 for the train set and 0.3383 for the test set.

I've tried other machine learning models like Gradient Boosting Regressor, Random forest regressor, decision tree regressor but they all have high mean square error.

I'm new to keras and deep learning. How do I solve the issue?

Is this amount of training data enough for the neural network? I can't get more data.

• If none of that is working, something might be wrong with your network architecture/code. If you post your code, probably it will make the question more specific and people will be able to help. – timkartar Feb 5 '20 at 3:27
• The fact that you're getting high loss for both neural net and other regression models, and a lowish r-squared from the training set might indicate that the features (X values) you're using only weakly explain the targets (y values). Particularly if even a GBDT model doesn't fit well – David Waterworth Feb 5 '20 at 3:54
• @timkartar I've edited the question to include code. – doofensmirtz Feb 5 '20 at 18:40
• @DavidWaterworth correlation and causal analysis between the features and the target variables suggest that the target might depend on the chosen input variables – doofensmirtz Feb 5 '20 at 18:42
• 1) Is the in-sample performance acceptable? You mention getting in-sample $R^2 = 0.5276$. If you got an out-of-sample $R^2$ around there, would that be good enough for what you want to do? 2) No, you probably don't have enough data. What's your parameter count? 3) The use of $R^2$ in nonlinear regression is controversial. – Dave Feb 5 '20 at 18:59

1. the network architecture above is a very strange choice. When you have only 6 input features, it is weird to have so much Dense layers stacked.

2. if network is overfitting, WHERE IS DROPOUT? Why not trying some regularizers, if the latter does not help?

3. +1 for David Waterworth - correlation/causal analysis is not everything yet. Does linear regression provide better R-square values?

4. what is output(target) variable range? Maybe it should be mapped/scaled to something reasonable? (I judge from loss values).

5. activation function and initializers are important too. Try using different values, rather than relu/linear and 'normal' initializer.

EDIT: yes, this should be enough data, if your data has only 6 inputs. However, you can try augmenting data too, if it makes sense and you can make reasonable assumptions in your case - sometimes it gives difference in the long run, even if in the beginning you think it does not work.

EDIT2: with specific datasets, neural network can get into local plateau (not minima however), where it does not escape. To test this hypothesis, you can set learning rate to small value and all initializers to generate small values too - then network may not go to this plateau suddenly, but goes to global minima instead.

EDIT3: increasing batch size leads to faster but poorer convergence on certain datasets. On a smaller network, batch size = 1 sometimes makes wonders.

• 1) what architecture do you suggest. I've tried 2) and 5). 3) Linear regression doesn't provide good r squared value. 4) Output target variable range was from 1-25000 initially. Now I tried to normalise the output column as well. Do you think that is a good idea? What other options do I have? Thank you for your answer – doofensmirtz Feb 13 '20 at 17:12
• It is hard to tell without a dataset. As follows from 1. and 2. - reduce number of Dense layers say to 4, and add Dropout layers between them, starting from small 0.05 dropout rate. – Emil Feb 13 '20 at 17:19
• In my practise, I used target normalisation, it helped sometimes. Not necessarily linearly, but square root, log function is good - depends on distribution. – Emil Feb 13 '20 at 22:24
• Try batch normalization and orthogonal, glorot_normal initialization too. – Emil Feb 13 '20 at 22:25
• Thank You @Emil for this useful info. – doofensmirtz Feb 14 '20 at 13:34