5
$\begingroup$

I am using Tensorflow to predict whether the given sentence is positive and negative. I have take 5000 samples of positive sentences and 5000 samples of negative sentences. 90% of the data I used it for training the neural network and rest 10% for testing.

Below is the parameter initialisation. Batch size = 100 Epochs = 30 Number of hidden layers = 3 Nodes in each hidden layer = 100 epochs = 30

I could see in each epoch the cost function is getting reduced reasonably. However the accuracy of the model on test set is poor (only 56%)

 Epoch  1 completed out of  30 loss :  22611.10902404785
 Epoch  2 completed out of  30 loss :  12377.467597961426
 Epoch  3 completed out of  30 loss :  8659.753067016602
 Epoch  4 completed out of  30 loss :  6678.618850708008
 Epoch  5 completed out of  30 loss :  5391.995906829834
 Epoch  6 completed out of  30 loss :  4476.406986236572
 Epoch  7 completed out of  30 loss :  3776.497922897339
 -------------------------------------------------------
 Epoch  25 completed out of  30 loss :  478.93606185913086
 Epoch  26 completed out of  30 loss :  450.8017848730087
 Epoch  27 completed out of  30 loss :  435.0913710594177
 Epoch  28 completed out of  30 loss :  452.10553523898125
 Epoch  29 completed out of  30 loss :  539.5199084281921
 Epoch  30 completed out of  30 loss :  685.9198244810104
 Accuracy of Train : 0.88155556
 Accuracy of Test : 0.524

Is there any parameter that can be tuned to increase the accuracy of the model considering the same number of data set.

$\endgroup$
  • $\begingroup$ How about to do regularization to simplify your fitting model? I believe it can improve your accuracy. $\endgroup$ – MinSama May 5 '18 at 10:46
  • $\begingroup$ @MinSama I will try to insert the regularization parameter and update here. Also I am looking for more samples of data as the below answers say. $\endgroup$ – deepguy May 11 '18 at 2:20
3
$\begingroup$

You need much more data.

Deep NN shines when you have excessive amounts of data.

With only a little bit if data it can easily overfit. The big difference between training and test performance shows that your network is overfitting badly. This is likely also because your network model has too much capacity (variables, nodes) compared to the amount of training data. A smaller network (fewer nodes) may overfit less.

$\endgroup$
  • $\begingroup$ I tried to implement the same with 1 hidden layer of 5 nodes, but still I got accuracy of 1 for both training and test. Have been doing lot of debug, wondering why 100% accuracy.! $\endgroup$ – deepguy May 6 '18 at 8:30
  • $\begingroup$ any suggestions ? $\endgroup$ – deepguy May 10 '18 at 19:49
  • $\begingroup$ Get much more data. 5000 is just way too little for text. $\endgroup$ – Has QUIT--Anony-Mousse May 10 '18 at 20:21
1
$\begingroup$

For increasng your accuracy the simplest thing to do in tensorflow is using Dropout technique. Try to use tf.nn.dropout. between your hidden layers. Do not use it for your first and last layers. For applying that, you can take a look at How to apply Drop Out in Tensorflow to improve the accuracy of neural network.

$\endgroup$
  • 1
    $\begingroup$ Thanks for the suggestion and link. I will try this method and update you. $\endgroup$ – deepguy May 5 '18 at 15:47
1
$\begingroup$

Whatever regularization technique you're using, if you keep training long enough, you will eventually overfit the training data, you need to keep track of the validation loss each epoch. at least use early stopping to stop the training process when the validation loss stops decreasing.

$\endgroup$
  • $\begingroup$ Thanks for the reply. I could see at epoch 27 loss has reached the minimum and started raising again. I think it needs to be stopped at epoch 27 right? $\endgroup$ – deepguy May 11 '18 at 2:23
  • $\begingroup$ The loss on the log is it from validation or training? if it's from training you have to further split the data into validation set. $\endgroup$ – Fadi Bakoura May 11 '18 at 3:31
  • $\begingroup$ The loss mentioned in the log is from the training data set generated in each epoch while training. $\endgroup$ – deepguy May 11 '18 at 4:06
  • $\begingroup$ When the training loss increases, it means the model has a divergence caused by a large learning rate. the thing is, when doing SGD, we are estimating the gradient. therefore when a noisy update is repeated (training too many epochs) the weights will be in a bad position far from any good local minimum. and the non-linearity activation functions are saturated. my suggestion to you is to read this paper: arxiv.org/abs/1206.5533 $\endgroup$ – Fadi Bakoura May 11 '18 at 8:12
1
$\begingroup$

This is simple case of overfitting.

To improve accuracy, I would suggest to do the following changes:

Since your 'x' variable are sentences, you can try to use Sequential model with one Embedding Layer and one LSTM layer:

from tensorflow.keras.layers import Dense, Embedding, LSTM
from tensorflow.keras.models import Sequential

model = Sequential()

model.add(Embedding(max_features, 32))
model.add(LSTM(32)

Add the last layer as Dense Layer, and the loss function as binary_crossentropy:

model.add(Dense(1, activation='sigmoid'))

model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])

Fit the model and store the history variable to find the optimal epochs:

history = model.fit(x_train, y_train, epochs=20, batch_size=128, validation_split=0.2)

Plot the following 2 curves: i) Validation loss vs epochs ii) Validation accuracy vs epochs

# Plotting Results
import matplotlib.pyplot as plt

acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(1, len(acc) + 1)

plt.plot(epochs, acc, 'b', label='Training acc')
plt.plot(epochs, val_acc, 'g', label='Validation acc')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')

plt.title('Training and validation accuracy')
plt.legend()
fig = plt.figure()
fig.savefig('acc.png')


plt.plot(epochs, loss, 'b', label='Training loss')
plt.plot(epochs, val_loss, 'g', label='Validation loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.title('Training and validation loss')

plt.legend()
plt.show()

After which you will get a graph something like these: enter image description here

enter image description here

For this specific case, we see that the optimal epochs is occurring at 12, So we need to again train with 12 epochs and test on test data.

model.fit(x_train, y_train, epochs=12, batch_size=128)
model.evaluate(x_test, y_test)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.