How to improve accuracy of deep neural networks

Question

I am using Tensorflow to predict whether the given sentence is positive and negative. I have take 5000 samples of positive sentences and 5000 samples of negative sentences. 90% of the data I used it for training the neural network and rest 10% for testing.

Below is the parameter initialisation. Batch size = 100 Epochs = 30 Number of hidden layers = 3 Nodes in each hidden layer = 100 epochs = 30

I could see in each epoch the cost function is getting reduced reasonably. However the accuracy of the model on test set is poor (only 56%)

 Epoch  1 completed out of  30 loss :  22611.10902404785
 Epoch  2 completed out of  30 loss :  12377.467597961426
 Epoch  3 completed out of  30 loss :  8659.753067016602
 Epoch  4 completed out of  30 loss :  6678.618850708008
 Epoch  5 completed out of  30 loss :  5391.995906829834
 Epoch  6 completed out of  30 loss :  4476.406986236572
 Epoch  7 completed out of  30 loss :  3776.497922897339
 -------------------------------------------------------
 Epoch  25 completed out of  30 loss :  478.93606185913086
 Epoch  26 completed out of  30 loss :  450.8017848730087
 Epoch  27 completed out of  30 loss :  435.0913710594177
 Epoch  28 completed out of  30 loss :  452.10553523898125
 Epoch  29 completed out of  30 loss :  539.5199084281921
 Epoch  30 completed out of  30 loss :  685.9198244810104
 Accuracy of Train : 0.88155556
 Accuracy of Test : 0.524

Is there any parameter that can be tuned to increase the accuracy of the model considering the same number of data set.

Answer 1

You need much more data.

Deep NN shines when you have excessive amounts of data.

With only a little bit if data it can easily overfit. The big difference between training and test performance shows that your network is overfitting badly. This is likely also because your network model has too much capacity (variables, nodes) compared to the amount of training data. A smaller network (fewer nodes) may overfit less.

Answer 2

For increasng your accuracy the simplest thing to do in tensorflow is using Dropout technique. Try to use tf.nn.dropout. between your hidden layers. Do not use it for your first and last layers. For applying that, you can take a look at How to apply Drop Out in Tensorflow to improve the accuracy of neural network.

Answer 3

Whatever regularization technique you're using, if you keep training long enough, you will eventually overfit the training data, you need to keep track of the validation loss each epoch. at least use early stopping to stop the training process when the validation loss stops decreasing.

Answer 4

This is simple case of overfitting.

To improve accuracy, I would suggest to do the following changes:

Since your 'x' variable are sentences, you can try to use Sequential model with one Embedding Layer and one LSTM layer:

from tensorflow.keras.layers import Dense, Embedding, LSTM
from tensorflow.keras.models import Sequential

model = Sequential()

model.add(Embedding(max_features, 32))
model.add(LSTM(32)

Add the last layer as Dense Layer, and the loss function as binary_crossentropy:

model.add(Dense(1, activation='sigmoid'))

model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])

Fit the model and store the history variable to find the optimal epochs:

history = model.fit(x_train, y_train, epochs=20, batch_size=128, validation_split=0.2)

Plot the following 2 curves: i) Validation loss vs epochs ii) Validation accuracy vs epochs

# Plotting Results
import matplotlib.pyplot as plt

acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(1, len(acc) + 1)

plt.plot(epochs, acc, 'b', label='Training acc')
plt.plot(epochs, val_acc, 'g', label='Validation acc')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')

plt.title('Training and validation accuracy')
plt.legend()
fig = plt.figure()
fig.savefig('acc.png')


plt.plot(epochs, loss, 'b', label='Training loss')
plt.plot(epochs, val_loss, 'g', label='Validation loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.title('Training and validation loss')

plt.legend()
plt.show()

After which you will get a graph something like these:

For this specific case, we see that the optimal epochs is occurring at 12, So we need to again train with 12 epochs and test on test data.

model.fit(x_train, y_train, epochs=12, batch_size=128)
model.evaluate(x_test, y_test)