I am using the BERT implementation in https://github.com/google-research/bert for feature extracting and I have noticed a weird behaviour which I was not expecting: if I execute the program twice on the same text, I get different results. I need to know if this is normal and why this happens in order to treat this fact in one or another way. Why is the reason for this? Aren't neural networks deterministic algorithms?
$\begingroup$
$\endgroup$
0
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I had the exact same problem using the pytorch implementation, until I realised I didn't set the model in eval mode. Hence, dropout was still activated. I guess this is also the source of your undeterministic behavior.
Fix using pytorch:
model.eval()
Using tensorflow with an Estimator, ensure you call estimator.evaluate or estimator.predict during testing. If you exported the model in some ways, then you should check the tensorflow documentation.
