Before I present my problem, please note that I am a newbie in deep learning and I am trying things for the first time. Most of my code/logic were adopted from various references in the internet.
Goal : Build a LSTM/CNN model to classify the IMDB reviews available in tensorflow datasets
Approach 1 : 1) LSTM based - train data - 45000 (10% validation split),test data - 5000 , accuracy > 95% , validation_accuracy > 85% , glove embeddings of size 100 was used Approach 2 : 1) CNN model - a) train data - 45000 , test_data - 5000 b) train data - 50% , test_data - 50% accuracy > 95% , validation_accuracy > 85%
Problem : Test_data accuracy doesn't go beyond 52% with both the approaches.Most of the code/references available out there use test_data during training. test_data wasn't part of my training.
Methodologies tried to increase test accuracy :
- Movie reviews length (padded) and max number of words in the vocabulary
- train test split ratio
- embeddings trainable=true/false
- with and without GLove word embeddings
My guess is there isn't enough training data. I need help on how to increase the test data accuracy.