1
$\begingroup$

I have small corpus max 150 text utterances, which is again distributed among 5 categories. To test I started with basic deep learning model, where I used word2vec embedding, added 1D convolution layer followed by 150 GRU units:

embedding_layer = Embedding(vocab_size, 300,weights=[embedding_matrix], input_length=max_length,trainable=True)
sequence_input = Input(shape=(max_length,), dtype='int32')
embedded_sequences = embedding_layer(sequence_input)

x = Conv1D(24, 4,activation='relu')(embedded_sequences)
x = GRU(150,kernel_regularizer=regularizers.l1(0.01),activation='tanh')(x)
x = Dropout(0.4)(x)

Training & validation loss:

training & validation loss As per this it's fits perfect, but when I give as is utterances to predict it's going to come other class.

Code for predicting the utterances:

encoded_doc = [[16, 7, 49, 50, 51]]
max_length = 25
padded_doc = pad_sequences(encoded_doc, maxlen=max_length, padding='post')
predictions = model.predict(padded_doc)
pre_class = model.predict(padded_doc)[0]
classes = np.argmax(predictions)

print('Predicted class: '+str(label_encoder.inverse_transform(classes))+' ## score: '+str(pre_class[classes]))

So, I changed it to LSTM where I increased the LSTM units to 250:

seed = 7
np.random.seed(seed)

embedding_layer = Embedding(vocab_size, 300,weights=[embedding_matrix], input_length=max_length,trainable=True)
sequence_input = Input(shape=(max_length,), dtype='int32')
embedded_sequences = embedding_layer(sequence_input)

x = Conv1D(24, 4,activation='relu')(embedded_sequences)
x = LSTM(250,kernel_regularizer=regularizers.l1(0.01),activation='tanh')(x)
x = Dropout(0.4)(x)

Train & Validation loss:

enter image description here

_________________________________________________________________ 
Layer (type)                 Output Shape              Param #   
================================================================= 
input_7 (InputLayer)         (None, 25)                0         
_________________________________________________________________                     
embedding_7 (Embedding)      (None, 25, 300)           31500     
_________________________________________________________________     
conv1d_7 (Conv1D)            (None, 22, 64)            76864     
_________________________________________________________________ 
lstm_4 (LSTM)                (None, 250)               315000    
_________________________________________________________________ 
dropout_7 (Dropout)          (None, 250)               0         
_________________________________________________________________ 
dense_7 (Dense)              (None, 5)                 1255      
================================================================= 
Total params: 424,619 Trainable params: 424,619 Non-trainable params:0
_________________________________________________________________ 
Train on 123 samples, validate on 31 samples Epoch 1/35 123/123
> [==============================] - 1s 8ms/step - loss: 22.8336 - acc:
> 0.2439 - val_loss: 18.5322 - val_acc: 0.3871 Epoch 2/35 123/123 [==============================] - 1s 7ms/step - loss: 16.5612 - acc:
> 0.2846 - val_loss: 12.7986 - val_acc: 0.4194 Epoch 3/35 123/123 [==============================] - 1s 7ms/step - loss: 11.0305 - acc:
> 0.3171 - val_loss: 7.7372 - val_acc: 0.5806 Epoch 4/35 123/123 [==============================] - 1s 7ms/step - loss: 6.5548 - acc:
> 0.4472 - val_loss: 4.1708 - val_acc: 0.5806 Epoch 5/35 123/123 [==============================] - 1s 7ms/step - loss: 3.2714 - acc:
> 0.5447 - val_loss: 1.9674 - val_acc: 0.5484 Epoch 6/35 123/123 [==============================] - 1s 7ms/step - loss: 1.6011 - acc:
> 0.5528 - val_loss: 1.2261 - val_acc: 0.5806 Epoch 7/35 123/123 [==============================] - 1s 7ms/step - loss: 2.0814 - acc:
> 0.4878 - val_loss: 1.9198 - val_acc: 0.5484 Epoch 8/35 123/123 [==============================] - 1s 7ms/step - loss: 1.7965 - acc:
> 0.4634 - val_loss: 1.2227 - val_acc: 0.5806 Epoch 9/35 123/123 [==============================] - 1s 7ms/step - loss: 1.4348 - acc:
> 0.5447 - val_loss: 1.2684 - val_acc: 0.6129 Epoch 10/35 123/123 [==============================] - 1s 7ms/step - loss: 1.3092 - acc:
> 0.5772 - val_loss: 1.0482 - val_acc: 0.7742 Epoch 11/35 123/123 [==============================] - 1s 7ms/step - loss: 1.2495 - acc:
> 0.6341 - val_loss: 1.0036 - val_acc: 0.7419 Epoch 12/35 123/123 [==============================] - 1s 7ms/step - loss: 1.1438 - acc:
> 0.7073 - val_loss: 0.9640 - val_acc: 0.7419 Epoch 13/35 123/123 [==============================] - 1s 7ms/step - loss: 0.8768 - acc:
> 0.9024 - val_loss: 0.4931 - val_acc: 1.0000 Epoch 14/35 123/123 [==============================] - 1s 7ms/step - loss: 0.5908 - acc:
> 0.9512 - val_loss: 2.2134 - val_acc: 0.6129 Epoch 15/35 123/123 [==============================] - 1s 7ms/step - loss: 1.4977 - acc:
> 0.6423 - val_loss: 1.1113 - val_acc: 0.5806 Epoch 16/35 123/123 [==============================] - 1s 7ms/step - loss: 1.1936 - acc:
> 0.5691 - val_loss: 1.0105 - val_acc: 0.5806 Epoch 17/35 123/123 [==============================] - 1s 7ms/step - loss: 1.1522 - acc:
> 0.4634 - val_loss: 1.0109 - val_acc: 0.5806 Epoch 18/35 123/123 [==============================] - 1s 7ms/step - loss: 1.0135 - acc:
> 0.6260 - val_loss: 2.5970 - val_acc: 0.1935 Epoch 19/35 123/123 [==============================] - 1s 7ms/step - loss: 1.4423 - acc:
> 0.6992 - val_loss: 1.3537 - val_acc: 0.6774 Epoch 20/35 123/123 [==============================] - 1s 7ms/step - loss: 1.4512 - acc:
> 0.5935 - val_loss: 1.1868 - val_acc: 0.6774 Epoch 21/35 123/123 [==============================] - 1s 7ms/step - loss: 1.2350 - acc:
> 0.5447 - val_loss: 1.0781 - val_acc: 0.5806 Epoch 22/35 123/123 [==============================] - 1s 7ms/step - loss: 1.1008 - acc:
> 0.6260 - val_loss: 0.9849 - val_acc: 0.9677 Epoch 23/35 123/123 [==============================] - 1s 7ms/step - loss: 0.9986 - acc:
> 0.6504 - val_loss: 0.8684 - val_acc: 0.9355 Epoch 24/35 123/123 [==============================] - 1s 7ms/step - loss: 1.3619 - acc:
> 0.7154 - val_loss: 1.4444 - val_acc: 0.6774 Epoch 25/35 123/123 [==============================] - 1s 7ms/step - loss: 1.5590 - acc:
> 0.7398 - val_loss: 1.5238 - val_acc: 0.5806 Epoch 26/35 123/123 [==============================] - 1s 7ms/step - loss: 1.1659 - acc:
> 0.8862 - val_loss: 0.8608 - val_acc: 1.0000 Epoch 27/35 123/123 [==============================] - 1s 7ms/step - loss: 0.8432 - acc:
> 0.9756 - val_loss: 0.6919 - val_acc: 1.0000 Epoch 28/35 123/123 [==============================] - 1s 8ms/step - loss: 0.7218 - acc:
> 0.9675 - val_loss: 0.6103 - val_acc: 1.0000 Epoch 29/35 123/123 [==============================] - 1s 7ms/step - loss: 0.6496 - acc:
> 0.9756 - val_loss: 0.5566 - val_acc: 1.0000 Epoch 30/35 123/123 [==============================] - 1s 7ms/step - loss: 0.5961 - acc:
> 0.9675 - val_loss: 0.5152 - val_acc: 1.0000 Epoch 31/35 123/123 [==============================] - 1s 7ms/step - loss: 0.5590 - acc:
> 0.9675 - val_loss: 0.4832 - val_acc: 1.0000 Epoch 32/35 123/123 [==============================] - 1s 9ms/step - loss: 0.5188 - acc:
> 0.9593 - val_loss: 0.4564 - val_acc: 1.0000 Epoch 33/35 123/123 [==============================] - 1s 10ms/step - loss: 0.4987 - acc:
> 0.9675 - val_loss: 0.4355 - val_acc: 1.0000 Epoch 34/35 123/123 [==============================] - 1s 7ms/step - loss: 0.4634 - acc:
> 0.9919 - val_loss: 0.4177 - val_acc: 1.0000 Epoch 35/35 123/123 [==============================] - 1s 7ms/step - loss: 0.4372 - acc:
> 1.0000 - val_loss: 0.4043 - val_acc: 1.0000 31/31 [==============================] - 0s 1ms/step Accuracy: 100.000000

I am not getting any clue where I am going wrong, even to avoid that I have hard coded the sentence tokens to predict. I wanted to understand is it over fitting or not cos at least by training & validation loss graph I don't think it's.

Please, let me know if you need any more information.


  • Epochs: 35
  • Batch size: 20
  • Shuffle = False
  • Validation split is on 20%

Thanks in advance.

$\endgroup$
1
$\begingroup$

Try increasing your batch size.

I think it could be because you specify a batch size of 35 and the validation will be tested on a batch_size of 32 by default. Testing a batch of 32 on the weights that were just reached the epoch might indeed lead to a slightly better average performance as all samples in the batch get the newest and best current weights.

If the samples in your train and validation set are extremely similar, I would expect the validation curve to always slightly beat the train score, given you use similar batch sizes (35 vs. 32).

You can see that the train and validation curves do indeed level out over time.

$\endgroup$
  • $\begingroup$ Thanks for your input. But I haven't set the batch size to 35 it was 20, and then I did few more tests like as you suggested to increase the batch size I did that. I made it to 60 so, my total utterances are 154, 80% of it will be 123 i.e 2 iterations to complete one epoch. But still I am getting same score & behavior. Even I tried after setting batch size to 10, 40 & 50. $\endgroup$ – Deep Oct 16 '18 at 18:05
  • $\begingroup$ I have increased the corpus size & kept the hyper para as is, the accuracy is better for as is utterances but no new issue is if I change one or two words from the utterances then it is significantly affecting the accuracy score, any idea why this happens? $\endgroup$ – Deep Oct 17 '18 at 11:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.