Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am consistently seeing higher validation loss when I train & evaluate a model on AWS GPU vs local CPU.
I am using the exact same train/eval datasets and the exact same Tensorflow code and configuration for both runs. Has anyone else experienced this before and know how to explain?
See screenshots below comparing GPU vs CPU loss for equivalent steps.
Since one could see there is no difference in the minmal loss value, this could well be an issue of differnce in floating point precision with respect to CPU and GPU. You should try to cross check unit operations in the intermediate layers(Considering its an ANN) and look for possible discrepencies to make sure it is a floating point precision variance.
