Retraining with the same data returns different accuracies

I am using TensorFlow to train a simple neural network (3 sequential dense layers). The problem is that the accuracy changes a lot every time I retrain it from scratch. I understand that, since the weights are initialized randomly, it may not always arrive at the exact same accuracy; but, I get a range of 4% for the accuracy on the test set.

This variation makes it impossible to check if different configurations of the network or different preprocessing steps for the data work better or worse because the configuration/preprocessing is a good/bad idea, or just because I got lucky/unlucky with the random initial weights.

This is an example of accuracies on 5 consecutive train+test's. The numbers are:

1. Accuracy on the train set
2. Accuracy on the validation split (20%)
3. Accuracy on the test set
4. MSE
        1.  2.  3.  4.
Run #1: 95  91  74  20
Run #2: 94  92  75  18
Run #3: 94  91  74  20
Run #4: 94  92  73  20
Run #5: 94  91  77  17


I also find it confusing that there is no correlation between the accuracies for training, validation, and test sets.

I have tried different configurations of the ANN, longer trainings, shorter trainings, bigger and smaller validation split, different optimizers...nothing seems to give me a more stable accuracy among re-trainings. The numbers I have posted here are the best I could get.

Is a 4% range something normal? Is there a way to avoid those sub-optimal trainings? Could it be a problem related to local minima?

• How are you choosing the threshold for classification ? – lcrmorin Jul 21 at 13:47
• @lcrmorin I'm not sure I understand what you mean. The activation in the last layer is a softmax. Then I just log the accuracy and MSE metrics from TensorFlow. – Mr. Goferito Jul 21 at 13:52
• What loss function are you using? – noe Jul 21 at 14:21
• @noe Mean Squared Error – Mr. Goferito Jul 21 at 14:22
• For classification problems, you should normally use categorical cross-entropy, not MSE. – noe Jul 21 at 14:53

I suggest to apply a nonrandom weight initialisation in order to see the impact of random initialization.

For instance, you can use the Nguyen-Widrow weight initialization.

def initnw(layer):
"""
Nguyen-Widrow initialization function

:Parameters:
layer: core.Layer object
Initialization layer
"""
ci = layer.ci
cn = layer.cn
w_fix = 0.7 * cn ** (1. / ci)
w_rand = np.random.rand(cn, ci) * 2 - 1
# Normalize
if ci == 1:
w_rand = w_rand / np.abs(w_rand)
else:
w_rand = np.sqrt(1. / np.square(w_rand).sum(axis=1).reshape(cn, 1)) * w_rand

w = w_fix * w_rand
b = np.array([0]) if cn == 1 else w_fix * np.linspace(-1, 1, cn) * np.sign(w[:, 0])

# Scaleble to inp_active
amin, amax  = layer.transf.inp_active
amin = -1 if amin == -np.Inf else amin
amax = 1 if amax == np.Inf else amax

x = 0.5 * (amax - amin)
y = 0.5 * (amax + amin)
w = x * w
b = x * b + y

# Scaleble to inp_minmax
minmax = layer.inp_minmax.copy()
minmax[np.isneginf(minmax)] = -1
minmax[np.isinf(minmax)] = 1

x = 2. / (minmax[:, 1] - minmax[:, 0])
y = 1. - minmax[:, 1] * x
w = w * x
b = np.dot(w, y) + b

layer.np['w'][:] = w
layer.np['b'][:] = b

return


On the other hand, remember that local minimum algorithms (Gradient Descent, Adam Optimizer, etc.) used to have some stochastic behavior, for example in the definition of the starting point, or specific noise parameters like epsilon.

• I gave this a try but it didn't help. The variation remains. I guess because there is still randomness because of the optimizer. – Mr. Goferito Jul 24 at 20:51
• At least you know where the randomness comes mostly from. Maybe you could try to modify an optimizer to remove the randomness? For example with a pre defined starting point. – Nicolas M Jul 25 at 6:49

A possible cause of the problem is that you are using the mean-squared error (MSE) as loss function for a classification problem.

Normally, for classification you would use categorical cross-entropy.

• I did a few more train+test's using Tensorflow's softmaxCrossEntropy loss function, and the result is pretty much the same in terms of accuracy achieved, and there is still a 4% variation in the accuracy for the test set (76% to 80%) – Mr. Goferito Jul 21 at 23:57
• I understand that you also removed the softmax activation in the last layer, correct? – noe Jul 22 at 9:05
• I just gave it a try removing the softmax activation. Now the variation is 3% instead of 4%. Is that meant to be acceptable? – Mr. Goferito Jul 22 at 14:13
• There is no standard for "acceptable" and, as far as I know, there are no studies of the variability of results based on different seeds for regression problems with neural networks. Also, if you want to go deeper in this matter, I would study the distribution of the results with many more seeds. – noe Jul 22 at 15:10

If you're getting 95% accuracy on training set, but only 75% on test set, this points to serious overfitting, which none of the measures you've listed are likely to address.

It's also suspicious that validation result are so close to training, but far from test. This often happens when you change validation set during training, meaning there's effectively no validation set at all. Or if you keep training over and over too many times until obtaining the desired accuracy on validation set, which is also a recipe for overfitting.

• "If you're getting 95% accuracy on training set, but only 75% on test set, this points to serious overfitting" - perhaps, but not necessarily. This may simply be the best result that can be obtained on the test set with that training set and/or model. I definitely would recommend trying to tweak the hyperparameters or stop training earlier to see whether that improves the result, but those percentages in themselves don't throw up an automatic red flag for me. Those options are both mentioned in the question, yet you say they won't address the problem, so how do you think one fixes overfitting? – NotThatGuy Jul 22 at 3:49
• @NotThatGuy sure, it may be possible that the training set is not entirely representative of the test data, and 75% is the best possible result. 90% validation result might be due to this, but also might point to "leaking" of validation data into training set. As for the OP's steps, "shorter trainings" he mentioned might prevent overfitting, but that depends. He never mentioned overfitting in the question, so it seemed to me that he might not be considering it. The easiest way to fix overfitting is a simpler model; maybe a simpler NN can only get 85% in training but similar results in test. – IMil Jul 22 at 4:51
• @NotThatGuy I have tried different shuffles of the entire data set, to check if the problem is that there is something special with the test set that doesn't generalize, but the results are the same. I also did trainings stopping earlier but I get worse accuracies (ie: train 88%, validation 86%, test 62%). Training as long as the validation accuracy keeps increasing also increases the accuracy on the test set, so I don't think I have an over-fitting problem. – Mr. Goferito Jul 22 at 11:56
• @IMil I also tried with a simpler model, even removing all hidden layers, what gave me: train 88%, validation 86%, test 84%. Bigger models need more time to train, but return better results. – Mr. Goferito Jul 22 at 12:01
• @Mr.Goferito Test set performance is generally the most important because that's (hopefully) most similar to what will happen in the real world, where you run your model on data you haven't seen before and didn't use for training. If a model is performing better on the test set (all else being equal, but especially when it's a simpler model), that's typically considered the better one. Although if the NN model performs best with no hidden layers, you might want to try simple logistic regression instead (or other model types), as that's quite similar, but without the NN complexity. – NotThatGuy Jul 22 at 12:29