0
$\begingroup$

I am building a convolutional neural network in Keras to try to predict binary classification of some text sequences.

cmodel = models.Sequential()
cmodel.add(layers.Conv1D(1, kernel_size=9, activation="relu", input_shape=(64,20)))
cmodel.add(layers.MaxPooling1D(5))
cmodel.add(layers.Conv1D(1, kernel_size=9, activation="relu"))
cmodel.add(layers.GlobalMaxPooling1D())
cmodel.add(layers.Dense(1, activation="sigmoid"))

cmodel.compile(
    optimizer='adam',
    loss='binary_crossentropy',
    metrics=['accuracy', keras.metrics.BinaryAccuracy(name="binary_accuracy", dtype=None, threshold=0.5)],
)
epochs = 2000
e = range(1, epochs + 1)
history = cmodel.fit(
    train_seqs_vec,
    train_labels,
    epochs=epochs,
    validation_split=0.2
)

I train my model for several thousand epochs, and I have noticed the following:

  1. The first time I trained my model, I was able to get both binary_accuracy (training set metric) and validation_binary_accuracy (validation set) to ~90%. I was very surprised by this level of accuracy, so I re-ran it. Since I have not been able to get past ~80%. Did I see 90% because the randomly initialized convolution kernel was 'better' and reached a global min (or at least a better local min?). If this is the case, do I just continue retraining until I find the best validation_binary_accuracy and use these weights?

  2. The difference between my accuracy and validation_accuracy is quite large, while the gap between the binary . How is Keras calculating accuracy... using a 0.5 threshhold? If so, why is it any different from binary_accuracy?

    loss: 0.3254 - accuracy: 0.8479 - binary_accuracy: 0.8020 - val_loss: 1.2466 - val_accuracy: 0.7015 - val_binary_accuracy: 0.8020
    

Something seems fishy to me that the binary accuracies are identical between train and test set -- and I doing something wrong? I am new to deep learning, so any advice is appreciated.

enter image description here

Code to generate the plot so you know I'm not plotting the same series on it!:

history_dict = history.history
acc_values = history_dict['binary_accuracy']
val_acc_values = history_dict['val_binary_accuracy']
plt.plot(e, acc_values, 'r-', label='Training binary accuracy')
plt.plot(e, val_acc_values, 'g-', label='Validation binary accuracy')
plt.title('Training and validation binary accuracy')
plt.xlabel('Epochs')
plt.ylabel('Binary Accuracy')
plt.legend()
plt.show()
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.