0
$\begingroup$

Trained the same model twice with the same dataset, the same parameters (Epochs, Batch Size, Learning rate, etc..). But both trained model shows different train as well as test accuracy on the same dataset. (code is same for two models)

Why both models show different accuracy?

  1. Train the model

    Test accuracy = 87.98%

  2. Again Train the model

    Test accuracy = 67.18%

$\endgroup$
4
  • $\begingroup$ If the training score is different then the 2nd model is not trained fully. What is the data size for both the set and what is happening in 3rd, 4th time? $\endgroup$ – 10xAI Dec 30 '20 at 15:02
  • $\begingroup$ Yes, the training score is different for both the trained models. Dataset size for training is 13,558 (9 class). I have tried 2 times. Let me check for the 3rd and 4th trials also. $\endgroup$ – Rina Dec 31 '20 at 3:36
  • $\begingroup$ At the 3rd time model's test accuracy: 85.68% & for 4th time: 76.38% $\endgroup$ – Rina Dec 31 '20 at 10:16
  • $\begingroup$ You could use random_state=42 when splitting the data. $\endgroup$ – idkfa.bfg2 Jan 29 at 23:13
0
$\begingroup$

Somewhere in the code there should be some parameter that is initialised randomly, this is usually called the random seed. It could be the different initialisation of your neural network weights that is affecting the results, or maybe the k-fold being different if done at random. The extent of the difference between the performance suggests that you might not have enough data to reliably learn a good model, or your model is underspecified and gets stuck at different local minima. Consider, changing the learning rate and other hyperparameters and see if the effect stays.

$\endgroup$
1
  • $\begingroup$ The random seed value is selected as 2 in the code. So, there are any such criteria for the selection of random seed value? $\endgroup$ – Rina Dec 30 '20 at 10:00
0
$\begingroup$

Often the initial training is slow, because the model is stuck in a part of parameter space with low gradients. Then it is essentially taking a random walk, until it reaches an attractor.

You can often get a better shaped loss surface by regularizing - then training is less reliant on luck.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.