1
$\begingroup$

I searched for other similar questions on Google and here, but couldn't find any. I'm consistently getting results of less quality when making deep learning with GPU, compared to CPU. Of course, it is much faster using GPU, but my results converge to lower quality on many different aspects: subjective quality of embedding, validation accuracy, loss, etc. It doesn't depend on the number of epochs, ie. results won't converge toward the same value: one is always inferior to the other. It's disconcerting when trying to achieve solid, reproducible SOTA results, where .1 or .2% is quite important (litteraly from top2 to top1).

I have a RTX2070 and a good CPU. Is it inherent to modern GPU ? Are approximations done using GPU that could explain this phenomenon ? Thanks for your inputs.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ I have the exact same GPU and I've never observed anything like that. You confirm you didn't change any hyperparameter between trainings? Meybe it's possible that your Loss function is extremely non-convex, and this makes your model very sensitive to random parameter initialization... idk I'm guessing what could be the reason. If that's the case you should restart training multiple times until you find it's working. $\endgroup$
    – Leevo
    Jun 8, 2020 at 8:02

1 Answer 1

Reset to default
3
$\begingroup$

In theory, CPU and GPU should reach same accuracy. But imo, generally people tend to use bigger batch size on GPU due to high parallelization. As you might know, in stochastic gradient descent a smaller batch size has a higher chance of reaching the global optimum. This could be a reason for the slight difference in accuracy.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.