3
$\begingroup$

After reinstalling Keras and Tensorflow on a Virtual Machine, I have noticed that models fail to train correctly (at the very least those I have checked). For example, taking this code from the Keras blog as a test case, instead of decreasing loss and final validation accuracy of ~0.8, loss quickly increased to ~7.5 and accuracy stayed at ~0.5.

It should be noted that when I tried it with older versions of Keras and Tensorflow (on the same computer), I did not observe this aberrant behavior.

I have repeated this experiment at three other setups in which the same behavior was observed - a two other computers using a VM, and a third computer, this time a laptop with no suitable GPU. On the other hand, trying this on two computers with GPUs, the training progressed as expected, with loss decreasing and accuracy increasing to the level advertised in the Keras blog and observed with the previous versions of Keras and Tensorflow while using a CPU.

From the evidence I can speculate that either Keras's or Tensorflow's interaction with the CPU was changed in a manner disrupting the training processes.

Has anyone here faced a similar problem? What can be done to solve this?

Improve this question
$\endgroup$
2
  • $\begingroup$ Did you solve your problem yet? I have pretty much the same issue, and it look like the answer below does not help me either. $\endgroup$
    – DCS
    May 23, 2018 at 12:06
  • $\begingroup$ I did not continue to peruse this problem as I procured a new GPU server and did not need to run on a CPU. If you find an answer, I would be happy if you were to share it here :) $\endgroup$
    – Lafayette
    May 23, 2018 at 14:23

1 Answer 1

Reset to default
3
$\begingroup$

I recently had a similar trouble with porting working models to tensorflow 1.7. I found that the initialization was the culprit. Adding model.add(Dense(2048, activation='relu', kernel_initializer=initializers.glorot_normal())) instead of model.add(Dense(2048, activation='relu')) fixed the problem.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the assist, but it did not work for me, loss is still at ~8 and acc at 0.5000. $\endgroup$
    – Lafayette
    Apr 24, 2018 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.