2
$\begingroup$

I need a model that is able to receive as input an image of a nutritional information chart and tell the level of sugar that the product has. It would be a 3-class classification problem (low if sugar is below 5g, medium if it's between 5 and 22.5g and high if it has more than 22.5g). I have prepared all the data and I have 16000 images in total. However, I'm not able to train a proper model with the data. I have tried a simple convolutional neural network of 3 convolutional layers, the pretrained inception resnet v2 from keras, and even an attentional model (Github). The result is always the same, an accuracy equal to the proportion of samples from the most common class. So these models are unable to solve the issue and just bet for the most likely.

What kind of network could be able to solve this problem? I have never dealt with networks that have to "read" and classify text.

$\endgroup$
2
  • $\begingroup$ I'm not expert in images but I think you should do it in 3 steps: 1) identify the specific sugar information from the image; 2) run OCR to obtain the value as text 3) extract the actual numeric value. Trying to train the model to do all three steps at once requires the model to be much more complex, whereas each of these steps independently is much easier. Additionally you could more easily fix problems in one part or the other. $\endgroup$ – Erwan Jan 31 at 21:19
  • $\begingroup$ Any examples how the images look like? $\endgroup$ – Peter Feb 4 at 18:22
1
$\begingroup$

Another option would be to apply an established Optical Character Recognition (OCR) system to the raw images. After converting the raw images to plain text, put the relevant data into a tabular dataframe. Once in a tabular dataframe, fit a deep learning or traditional machine learning model can be fit.

It can be tempting to build an end-to-end deep learning system. However, some problems are best solved by combination of subsystems. Those subsystems could include human expertise, rules, traditional machine learning, and just a small amount of deep learning.

$\endgroup$
1
  • $\begingroup$ Agreed, the more prior knowledge added to the ML pipeline the easier it becomes for the models to learn. OCR would be a step in the right direction and then maybe some error correction mechanism. $\endgroup$ – bonfab Feb 4 at 12:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.