0
$\begingroup$

Trained the same model twice with the same dataset, the same parameters (Epochs, Batch Size, Learning rate, etc..). But both trained model shows different train as well as test accuracy on the same dataset. (code is same for two models)

Why both models show different accuracy?

  1. Train the model

    Test accuracy = 87.98%

  2. Again Train the model

    Test accuracy = 67.18%

$\endgroup$
5
  • $\begingroup$ If the training score is different then the 2nd model is not trained fully. What is the data size for both the set and what is happening in 3rd, 4th time? $\endgroup$
    – 10xAI
    Dec 30, 2020 at 15:02
  • $\begingroup$ Yes, the training score is different for both the trained models. Dataset size for training is 13,558 (9 class). I have tried 2 times. Let me check for the 3rd and 4th trials also. $\endgroup$
    – Rina
    Dec 31, 2020 at 3:36
  • $\begingroup$ At the 3rd time model's test accuracy: 85.68% & for 4th time: 76.38% $\endgroup$
    – Rina
    Dec 31, 2020 at 10:16
  • $\begingroup$ You could use random_state=42 when splitting the data. $\endgroup$
    – idkfa.bfg2
    Jan 29, 2021 at 23:13
  • $\begingroup$ If you share the code you are using it will be more easy for us to help. $\endgroup$
    – hH1sG0n3
    Nov 30, 2021 at 13:11

2 Answers 2

0
$\begingroup$

Somewhere in the code there should be some parameter that is initialised randomly, this is usually called the random seed. It could be the different initialisation of your neural network weights that is affecting the results, or maybe the k-fold being different if done at random. The extent of the difference between the performance suggests that you might not have enough data to reliably learn a good model, or your model is underspecified and gets stuck at different local minima. Consider, changing the learning rate and other hyperparameters and see if the effect stays.

$\endgroup$
1
  • $\begingroup$ The random seed value is selected as 2 in the code. So, there are any such criteria for the selection of random seed value? $\endgroup$
    – Rina
    Dec 30, 2020 at 10:00
0
$\begingroup$

Often the initial training is slow, because the model is stuck in a part of parameter space with low gradients. Then it is essentially taking a random walk, until it reaches an attractor.

You can often get a better shaped loss surface by regularizing - then training is less reliant on luck.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.