Precision, recall and accuracy metrics significantly different between training/validation and actual predictions

Question

I have two sequential models built with Keras that train on data from a CSV file. This is how they are built

# First model

model = Sequential()
model.add(Input(shape=X_1.shape))
model.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))
model.add(Dense(64, activation='relu', kernel_initializer='he_uniform'))
model.add(Dropout(0.7))
model.add(Dense(32, activation='relu', kernel_initializer='he_uniform'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
              optimizer=Adam(learning_rate=0.001),
              metrics=[recall, prec, ba])

# Second Model

model_2 = Sequential()
model_2.add(Input(shape=X_1.shape))
model_2.add(Dense(64, activation='relu', kernel_initializer='he_uniform'))
model_2.add(Dropout(0.5))
model_2.add(Dense(32, activation='relu', kernel_initializer='he_uniform'))
model_2.add(Dense(1, activation='sigmoid'))
model_2.compile(loss='binary_crossentropy',
              optimizer=Adam(learning_rate=0.001),
              metrics=[recall, prec, ba])

My dataset is split in half, one for training/validation and one for testing, also I have about 47k rows of data across 19 columns. Most of them are categorical, just one is numerical, all of the categorical data is one hot encoded and the numerical is normalized using MinMaxScaler, When training the model I use the built-in metrics from Keras for Recall, Precision, Accuracy and I get decent numbers of above 0.7 for these. I calculate the F1 score of my results manually.

I then test the trained model on the resting data (the second half of the dataset) and calculate the metrics using Sklearn as I wasn't able to find some built-in functions for that in Keras. I use the below code.

# predict crisp classes for test set
yhat_classes = (model.predict(X_2) > 0.5).astype("int32")

# Calculate F1 score
accuracy = accuracy_score(y_2, yhat_classes)
precision = precision_score(y_2, yhat_classes)
recall = recall_score(y_2, yhat_classes)
f1 = f1_score(y_2, yhat_classes)

On these metrics, I get considerably lower numbers across all of them. Again my validation numbers from the .fit method of the model are exactly what I need. I also use EarlyStopping on my model, watching val_recall when training.

Why are my resulting numbers from the prediction so different than training and how can I improve them? Is there a difference between how Sklearn and Keras calculate Precision and Recall?

Answer 1

Your model is overfitted. It means you are like "memorizing" from training data, when you should "learn". A good model must have similar scores for Training and validation data, but it's not your case, scores are super different. To avoid that:

• Use more data to train. Why you use only 50% for training? I will recommend you use about 90% almost. Anyway, you will be keeping around 5000 row to test your model, more than enough. With more data, more difficult to overfit. Moreover you can use techniques like data augmentation to obtain more data.
• Increase complexity of the model. Adding more layers or convolutions.
• You can try also to use some extra techniques like drop out or regularization.
• Decrease the value of learning rate, maybe you can improve your model with this step.

If you use a combination of these point I commented, you should see an improvement on your validations scores.