High model accuracy vs very low validation accuarcy

Question

I'm building a sentiment analysis program in python using Keras Sequential model for deep learning

my data is 20,000 tweets:

• positive tweets: 9152 tweets
• negative tweets: 10849 tweets

I wrote a sequential model script to make the binary classification as follows:

model=Sequential()
model.add(Embedding(vocab_size, 100, input_length=max_words))
model.add(Conv1D(filters=32, kernel_size=3, padding='same', activation='relu'))
model.add(MaxPooling1D(pool_size=2))
model.add(Flatten())
model.add(Dense(250, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# Fit the model

print(model.summary())
history=model.fit(X_train[train], y1[train], validation_split=0.30,epochs=2, batch_size=128,verbose=2)

however I get very strange results! The model accuracy is almost perfect (>90) whereas the validation accuracy is very low (<1) (shown bellow)

Train on 9417 samples, validate on 4036 samples
 Epoch 1/2
- 13s - loss: 0.5478 - acc: 0.7133 - val_loss: 3.6157 - val_acc: 0.0243
 Epoch 2/2
- 11s - loss: 0.2287 - acc: 0.8995 - val_loss: 5.4746 - val_acc: 0.0339

I tried to increase the number of epoch, and it only increases the model accuracy and lowers the validation accuracy

Any advice on how to overcome this issue?

Update:

this is how I handle my data

#read training data
pos_file=open('pos2.txt', 'r', encoding="Latin-1")
 neg_file=open('neg3.txt', 'r', encoding="Latin-1")
# Load data from files
pos = list(pos_file.readlines())
neg = list(neg_file.readlines()) 
x = pos + neg
docs = numpy.array(x)
#read Testing Data
pos_test=open('posTest2.txt', 'r',encoding="Latin-1")
posT = list(pos_test.readlines())
neg_test=open('negTest2.txt', 'r',encoding="Latin-1")
negT = list(neg_test.readlines())
xTest = posT + negT
total2 = numpy.array(xTest)

CombinedDocs=numpy.append(total2,docs)

# Generate labels
positive_labels = [1 for _ in pos]
negative_labels = [0 for _ in neg]
labels = numpy.concatenate([positive_labels, negative_labels], 0)

# prepare tokenizer
t = Tokenizer()
t.fit_on_texts(CombinedDocs)
vocab_size = len(t.word_index) + 1
# integer encode the documents
encoded_docs = t.texts_to_sequences(docs)
#print(encoded_docs)

# pad documents to a max length of 140 words
max_length = 140
padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')

Here I used Google public word2vec

# load the whole embedding into memory
embeddings_index = dict()
f = open('Google28.bin',encoding="latin-1")
for line in f:
values = line.split()
word = values[0]
coefs = asarray(values[1:], dtype='str')
embeddings_index[word] = coefs
f.close()

# create a weight matrix for words in training docs
embedding_matrix = zeros((vocab_size, 100))

for word, i in t.word_index.items():
embedding_vector = embeddings_index.get(word)
if embedding_vector is not None:
    embedding_matrix[i] = embedding_vector


#Convert to numpy
NewTraining=numpy.array(padded_docs)
NewLabels=numpy.array(labels)
encoded_docs2 = t.texts_to_sequences(total2)

# pad documents to a max length of 140 words

padded_docs2 = pad_sequences(encoded_docs2, maxlen=max_length, padding='post')


# Generate labels
positive_labels2 = [1 for _ in posT]
negative_labels2 = [0 for _ in negT]
yTest = numpy.concatenate([positive_labels2, negative_labels2], 0)
NewTesting=numpy.array(padded_docs2)
NewLabelsTsting=numpy.array(yTest)

Answer 1

You should try to shuffle all of your data and split them to the train and test and valid set then train again.

Answer 2

When a machine learning model has high training accuracy and very low validation then this case is probably known as over-fitting. The reasons for this can be as follows:

1. The hypothesis function you are using is too complex that your model perfectly fits the training data but fails to do on test/validation data.
2. The number of learning parameters in your model is way too big that instead of generalizing the examples , your model learns those examples and hence the model performs badly on test/validation data.

To solve the above problems a number of solutions can be tried depending on your dataset:

1. Use a simple cost and loss function.
2. Use regulation which helps in reducing over-fitting i.e Dropout.
3. Reduce the number of learning parameters in your model.

These are the 3 solutions that are most likely to improve the validation accuracy of your model and still if these don't work check your inputs whether they have the right shapes and sizes.

Answer 3

It seems that with validation split, validation accuracy is not working properly. Instead of using validation split in fit function of your model, try splitting your training data into train data and validate data before fit function and then feed the validation data in the feed function like this.

Instead of doing this

history=model.fit(X_train[train], y1[train], validation_split=0.30,epochs=2, batch_size=128,verbose=2)

Split your train data into validation and train data by any method, and then say your validation data is (X_val,Y_val), then replace the above line of code with this one:

history=model.fit(X_train[train], y1[train], validation_data=(X_val,Y_val),epochs=2, batch_size=128,verbose=2)

Answer 4

I had the same condition: High acc and low vad_acc.

It was because the parameter of Keras.model.fit, validation_split.

This will separate the last section of data as validation data. Therefore, if your data was in order, your validity data will be in the same case. Try to shuffle the training data.

Answer 5

It may be an imbalanced dataset problem during training though your number doesn't indicate. Try judiciously resampling with smote (oversampling) if the dataset is small or try any undersampling if the dataset is huge.