2
$\begingroup$

I need a model that is able to receive as input an image of a nutritional information chart and tell the level of sugar that the product has. It would be a 3-class classification problem (low if sugar is below 5g, medium if it's between 5 and 22.5g and high if it has more than 22.5g). I have prepared all the data and I have 16000 images in total. However, I'm not able to train a proper model with the data. I have tried a simple convolutional neural network of 3 convolutional layers, the pretrained inception resnet v2 from keras, and even an attentional model (Github). The result is always the same, an accuracy equal to the proportion of samples from the most common class. So these models are unable to solve the issue and just bet for the most likely.

What kind of network could be able to solve this problem? I have never dealt with networks that have to "read" and classify text.

$\endgroup$
2
  • $\begingroup$ I'm not expert in images but I think you should do it in 3 steps: 1) identify the specific sugar information from the image; 2) run OCR to obtain the value as text 3) extract the actual numeric value. Trying to train the model to do all three steps at once requires the model to be much more complex, whereas each of these steps independently is much easier. Additionally you could more easily fix problems in one part or the other. $\endgroup$
    – Erwan
    Jan 31, 2021 at 21:19
  • $\begingroup$ Any examples how the images look like? $\endgroup$
    – Peter
    Feb 4, 2021 at 18:22

1 Answer 1

1
$\begingroup$

Another option would be to apply an established Optical Character Recognition (OCR) system to the raw images. After converting the raw images to plain text, put the relevant data into a tabular dataframe. Once in a tabular dataframe, fit a deep learning or traditional machine learning model can be fit.

It can be tempting to build an end-to-end deep learning system. However, some problems are best solved by combination of subsystems. Those subsystems could include human expertise, rules, traditional machine learning, and just a small amount of deep learning.

$\endgroup$
1
  • $\begingroup$ Agreed, the more prior knowledge added to the ML pipeline the easier it becomes for the models to learn. OCR would be a step in the right direction and then maybe some error correction mechanism. $\endgroup$
    – bonfab
    Feb 4, 2021 at 12:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.