I need a model that is able to receive as input an image of a nutritional information chart and tell the level of sugar that the product has. It would be a 3-class classification problem (low if sugar is below 5g, medium if it's between 5 and 22.5g and high if it has more than 22.5g). I have prepared all the data and I have 16000 images in total. However, I'm not able to train a proper model with the data. I have tried a simple convolutional neural network of 3 convolutional layers, the pretrained inception resnet v2 from keras, and even an attentional model (Github). The result is always the same, an accuracy equal to the proportion of samples from the most common class. So these models are unable to solve the issue and just bet for the most likely.
What kind of network could be able to solve this problem? I have never dealt with networks that have to "read" and classify text.