0
$\begingroup$

I am consistently seeing higher validation loss when I train & evaluate a model on AWS GPU vs local CPU.

I am using the exact same train/eval datasets and the exact same Tensorflow code and configuration for both runs. Has anyone else experienced this before and know how to explain?

See screenshots below comparing GPU vs CPU loss for equivalent steps.

CPU Validation Loss

GPU Validation Loss

$\endgroup$
1
$\begingroup$

Since one could see there is no difference in the minmal loss value, this could well be an issue of differnce in floating point precision with respect to CPU and GPU. You should try to cross check unit operations in the intermediate layers(Considering its an ANN) and look for possible discrepencies to make sure it is a floating point precision variance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.