One thing I want to mention here. What kind of loss function you are using? From your results I deduce that you are using cross entropy with the parameter from_logits = True (that would explain the mentioned phenomenon), if you are with Keras, and you have the option from_logits = True, set it to false. I also recommend using label_smoothing=0.1 or more (depending on what you need). I leave you the link to the tensorflow cross entropy documentation if this is your case. https://www.tensorflow.org/api_docs/python/tf/keras/losses/CategoricalCrossentropy

One thing I want to mention here. What kind of loss function you are using? From your results, I deduce that you are using cross entropy with the parameter from_logits = True (that would explain the mentioned phenomenon) if you are with Keras, and you have the option from_logits = True, set it to false. I also recommend using label_smoothing = 0.1 or more (depending on what you need). I leave you the link to the TensorFlow cross entropy documentation if this is your case.