0
$\begingroup$

I have an NLP neural network that I have developed with Keras for multi-label classification.

I have fit the model several times and save the best results (via best validation accuracy score) after each set of epochs completes. All of my saved models are in the 96%+ validation accuracy score (according to Keras).

However, when I run these models against real-world data where I also know the result (e.g. effectively a second round of validation) one model in particular outperforms the rest. I can take the champion model (96.29% validation accuracy) and put it up against another model (with something like 96.18% validation accuracy) and the champion model can achieve 90%+ accuracy in the second round of validation while the other model - or any other model - will do nowhere near that. This one model will achieve a minimum 8% accuracy above all other models.

I have double-checked my methodology and I'm nearly positive that all models are being created with the same code and process.

Should I be concerned that this one particular model outperforms the rest? Does it indicate anything in particular in my overall methodology?

Improve this question
$\endgroup$
2
  • $\begingroup$ How do you feed text to classifier? $\endgroup$
    – Piotr Rarus
    Commented Dec 3, 2019 at 12:27
  • $\begingroup$ Real nice opportunity for investigation if you have the time. The first question, which maybe you can answer and maybe you can't, is: how does the second round data differ from the first? There are pretty clearly some differences in the distributions. $\endgroup$
    – Matthew
    Commented Apr 1, 2020 at 18:38

1 Answer 1

Reset to default
0
$\begingroup$

Maybe I did not get the question but all looks fine. This is how you do model selection. You have several models (either same algorithm with different parameters or different algorithms. does not matter) and then you perform cross validation to get the best model according to empirical errors coming from validation set. The best model wins the game and is chosen. Everything seems to be right.

Improve this answer
$\endgroup$
7
  • $\begingroup$ To be clear, these are all the same NN just run at different times throughout the day. All parameters are equal. So, really the only difference (I think) would be the random mix that Keras creates for each training epoch. No other parameters/processes were changed in between runs $\endgroup$
    – I_Play_With_Data
    Commented Nov 7, 2018 at 14:31
  • $\begingroup$ If Keras does the splitting for you then be sure it uses proper shuffling techniques to keep the results statistically significant. If you do it yourself then try to shuffle data and evaluate each model n times and see the mean and std of errors. That tells you what is the best model. If all models were literally the same then you have only one model and the empirical error is the mean of all. see this answer and the comment datascience.stackexchange.com/a/40862/8878 $\endgroup$
    – Kasra Manshaei
    Commented Nov 7, 2018 at 14:44
  • 2
    $\begingroup$ I do shuffle the data upon load before running all my epochs $\endgroup$
    – I_Play_With_Data
    Commented Nov 7, 2018 at 14:47
  • 1
    $\begingroup$ Yes. and one shuffle is just by chance well-separated (if it's a classification task). And anyways, you are not choosing any model as all of them are the same. Put the mean of all obtained errors in one basket and try "Other Models" (e.g. a NN with another architecture of layers) and see their errors as well. Then you can say which model is the best. So far there is no model"s" but just model. And it does not tell you anything $\endgroup$
    – Kasra Manshaei
    Commented Nov 7, 2018 at 14:49
  • 1
    $\begingroup$ While the models are all the same as @KasraManshaei says, I am assuming that the OP is saving the weights and biases from the best performing training session, and using these for subsequent prediction. There is nothing wrong with this. The random starting point for that model may just have been closer to the global minimum, thus avoiding some local minimum that prevents other models from performing as well. The only problem I can see with this is that should you need to, you may never be able to replicate the performance if you are forced to retrain. $\endgroup$
    – Skiddles
    Commented Nov 7, 2018 at 21:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.