I'm using keras to train a binary classifier neural network. To shuffle the training data I am using shuffle function from scikit-learn.
I observe that for some shuffle_random_state (seed for shuffle()), the network gives really good results (~86% accuracy) while on others not so much (~75% accuracy). So i run the model for 1-20 shuffle_random_states and choose the random_state which gives the best accuracy for production model.
I was wondering if this is a good approach and with those good shuffle_random_state the network is actually learning better?
$\begingroup$
$\endgroup$
4
-
$\begingroup$ The accuracy you are mentioning, is it on validation split or? If so, what is the accuracy on training split? $\endgroup$– Antonio JurićFeb 18, 2019 at 8:56
-
$\begingroup$ Mentioned accuracy is on validation split $\endgroup$– Chirag GuptaFeb 18, 2019 at 8:57
-
$\begingroup$ What is the accuracy on training split in those two cases? $\endgroup$– Antonio JurićFeb 18, 2019 at 8:58
-
$\begingroup$ Training loss and accuracy is almost the same in both cases. Goes till 100% if kept training. The rate of increase is also almost same for both cases (for training data) $\endgroup$– Chirag GuptaFeb 18, 2019 at 9:08
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
If this is split is a train/validation split (not a hold out test set) then you should be doing cross-validation. You are going to be overly optimistic about the performance of your model for this set of features and hyperparameters if you try to split it "just right". Cross-validation will give you a more accurate portrayal regardless of your split. If this is for a train/test split (test being a hold out test set), this is a very bad practice, since you are informing your decision on how to make the split based on the performance of the test set.
