Which neural network is better?

Question

MNIST dataset with 60 000 training samples and 10 000 test samples.
Neural network #1. Accuracy on the training set: 99.53%. Accuracy on the test set: 99.31%.
Neural network #2. Accuracy on the training set: 100.0%. Accuracy on the test set: 99.19%.
Which neural network is better if other parameters are unknown?
I have seen how many studies focus on accuracy on a test set, and rarely write about accuracy on a training set.

The first neural network is better in accuracy on the test set, but worse in accuracy on the training set.
Would you say that unlearned training samples can be bad for testing a different test set?

I have an idea to compare for overall accuracy:
(99.53% * 60000 + 99.31% * 10000) / (60000 + 10000) = 99.499%
(100.0% * 60000 + 99.19% * 10000) / (60000 + 10000) = 99.884%
Or it could be a weighted multiplication.
But I'm not sure about that.
What do you think about this?

Answer 1

When you train a neural network, you usually use 3 sets: one for training, one for development, one for testing.

• Your training set is here for (obviously) training your model: the performance of your model on your training set reflects how well your model learnt "by heart" what you showed it.

• Your development set is used jointly during training: your model does not see this data, but monitoring its accuracy on said set allows you to stop the training before overfitting.

  • Overfitting is when your model learns on your training set so well it can no longer generalise what it learned to new data, but can only reproduce what it has seen strictly. In general, you do not want your models to have this behavior, and you therefore stop the training at the inflexion point where your training accuracy increases while your development accuracy starts decreasing. (This is what you observe for your model 2: its training accuracy is higher, but dev accuracy lower)

• Your test set is only used at the end of your training to check, on yet a new set of data, that your model can still generalize. You NEVER stop your model training based on the test set, because it would introduce a bias.

TLDR. The only accuracy reported in the papers is the one on the test set. In most cases, how well the model learned the data "by heart" is not important (training accuracy) nor is how you chose to stop it before overfitting (dev accuracy). What is interesting is how well your model can generalize on completely new data, used neither for training nor for monitoring training (test accuracy).

Answer 2

The network is able to overfit the dataset, given enough epochs. So even if it performs great it doesn't mean that it will behave this way with unknown data.

Assuming that the accuracy is the right metric for your problem, you have to take into account only the results on the test set. These show the performance of your model in unknown data.

This is the reason you haven't seen any studies focusing on the training set.

Check out this article for a more in depth explanation of overfitting

Editing just to add that, as far as I know most studies will report results for a test set that is created by splitting the original dataset. Using more datasets with slightly different features and testing there, could also indicate how well your model is able to generalize. That is because most datasets are actually not representative of the whole problem domain.

For example, if you train a hate speech model with twitter posts, which were collected using specific hashtags (keywords for events etc.), it might be useful to also test the same trained model in other twitter hate speech test sets that might've been collected in a different way (different events, different time period etc). In my experience, this is something that makes a stronger case for your model and also guides you to find the best combination of both model architecture and dataset to train it on, in order for the model to approximate better the whole domain of expertise.

Answer 3

Based on the limited data you presented, Neural Network #1 is better.

The metric (accuracy) for both the training and test set are comparable, whereas the second network its overfitted for two reasons:

• Training metric is higher than test
• Training metric is 100%!!!

In the general case (new or old unseen data) the overfitted model is likely to perform less well.

What you shouldn't do is combine scores from the training and test set to generate some weird total.