I've been training on a local machine with Windows 11 (Version 10.0.22) with a 3070 Ti and recently have been able to access a DGX (Ubuntu 20.04.3) with 4 x V100s.

Despite numerous re-runs the model accuracy always improves on my local machine (30+ cases supporting this). Alternatively when I run that same script on the DGX it just bounces from 4.9-5.1% on both train and val acc (20 classes so this is random). No matter how many epochs this doesn't change and I've also tried this atleast 30+ times.

Things I've tried:

  1. Settings TF and NP random seeds. This does produce repeatable results however they still conform to the above issue regardless what the seed is, even if the same.

  2. Keras, Tensorflow-gpu, tensorflow, cudnn and cudatoolkit versions: I'd tell you what versions I'm using but I've basically created conda envs on both machines for every compatible configuration from TF2.0+. No change in outcome.

  3. Running both CPU only and disabling distribution strategy: I completely took the GPUs out of the equation and ran both purely on CPU. Again the local learns well (gets to 90%+ val acc) and the DGX hovers at 5%.

  4. Switched optimizers and loss functions

  5. Started with a model on the local machine (output .h5) and after 40% acc switching it over to the DGX to resume. Immediately shows ~5% acc and hangs around there.

  6. Cast the dataset DTYPE to a variety of precisions and even cast ints as floats.

  7. Different network models entirely and whole python scripts.

I'm honestly out of ideas, I think i've eliminated TF/Cudnn/cudatoolkit versions out as I've tried many versions, matched the machines and ran purely CPU. I think I've also eliminated the model/script as I've taken random models off github and notice a similar phenomena.

Could there be some fundamental issue that's explaining this difference? It's interesting to note the accuracy isn't locked at an exact value, rather just oscillates around random 5% so it appears backprop is occuring.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.