I am building a convolutional neural network in Keras to try to predict binary classification of some text sequences.
cmodel = models.Sequential() cmodel.add(layers.Conv1D(1, kernel_size=9, activation="relu", input_shape=(64,20))) cmodel.add(layers.MaxPooling1D(5)) cmodel.add(layers.Conv1D(1, kernel_size=9, activation="relu")) cmodel.add(layers.GlobalMaxPooling1D()) cmodel.add(layers.Dense(1, activation="sigmoid")) cmodel.compile( optimizer='adam', loss='binary_crossentropy', metrics=['accuracy', keras.metrics.BinaryAccuracy(name="binary_accuracy", dtype=None, threshold=0.5)], ) epochs = 2000 e = range(1, epochs + 1) history = cmodel.fit( train_seqs_vec, train_labels, epochs=epochs, validation_split=0.2 )
I train my model for several thousand epochs, and I have noticed the following:
The first time I trained my model, I was able to get both
binary_accuracy(training set metric) and
validation_binary_accuracy(validation set) to ~90%. I was very surprised by this level of accuracy, so I re-ran it. Since I have not been able to get past ~80%. Did I see 90% because the randomly initialized convolution kernel was 'better' and reached a global min (or at least a better local min?). If this is the case, do I just continue retraining until I find the best
validation_binary_accuracyand use these weights?
The difference between my
validation_accuracyis quite large, while the gap between the binary . How is Keras calculating
accuracy... using a 0.5 threshhold? If so, why is it any different from
loss: 0.3254 - accuracy: 0.8479 - binary_accuracy: 0.8020 - val_loss: 1.2466 - val_accuracy: 0.7015 - val_binary_accuracy: 0.8020
Something seems fishy to me that the binary accuracies are identical between train and test set -- and I doing something wrong? I am new to deep learning, so any advice is appreciated.
Code to generate the plot so you know I'm not plotting the same series on it!:
history_dict = history.history acc_values = history_dict['binary_accuracy'] val_acc_values = history_dict['val_binary_accuracy'] plt.plot(e, acc_values, 'r-', label='Training binary accuracy') plt.plot(e, val_acc_values, 'g-', label='Validation binary accuracy') plt.title('Training and validation binary accuracy') plt.xlabel('Epochs') plt.ylabel('Binary Accuracy') plt.legend() plt.show()