0
$\begingroup$

This is an issue I've been encountering before and I was wondering what can be potential causes for this. Occasionally, training of an identical setup LSTM model ( using Keras ), on the same training data, results in a much greater achieved validation accuracy.

E.g., a common scenario would be out of 10 training runs, it'd achieve a validation accuracy of 63% 8 out of 10 times, and 79% 2 out of 10 times. Out of all independent training runs it once even achieved a validation accuracy of 91%.

I know this can be caused by randomness in the neural network, I have (20%) dropout introduced and by default the seeds are randomly initialized by keras. However, is there a way I can "ensure" my model will find its best solution in one training? Or at least make it more reliable? Because it seems to indicate that my model gets stuck in local optima as it doesn't always find better solutions, even though it aparantly is able to find them in other runs.

Setting the seed to a constant value would not solve the issue because it will just force it to just find that 1 solution that corresponds with that specific seed, which may not be the best it's able to find in general.

Currently, I'm just training the model n times to assess its performance, analyzing its average and highest validation accuracy obtained. However this slows down my experiments in finding the best model a lot, and also the reliability is unknown.

So in short, how can I improve my model's ability to find its best solution in one training? Or what could be potential causes for it being inconsistent in doing so?

Any insights on this would be much appreciated!

$\endgroup$

1 Answer 1

1
$\begingroup$

In general, if there is a lot of difference in the results, it is usually due to a lack of data because it is difficult for the model to generalize patterns.

In addition to that, optimized parameters setups (learning rates, dropout, weights initialization, ...) could improve greatly the results consistency.

https://medium.com/geekculture/10-hyperparameters-to-keep-an-eye-on-for-your-lstm-model-and-other-tips-f0ff5b63fcd4

This could be done thanks to genetic algorithms that explores many hyper parameters and progressively improve them.

https://github.com/sunan93/Optimizing-RNN-parameters-using-Genetic-Algorithms

Keep in mind that having a too high accuracy could be an overfitting situation and could result bad in production. The right accuracy is around 85-90% but it depends on the data and the result in production. If you have a model with 95% accuracy in test and production, it should be kept.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.