we are trying to build a model to classify different types of documents as the first step in our pipeline (final goal is to read all the text). Currently we use ImageNet to extract the features and then pass it through a custom model which includes an LSTM layer and a couple of FC layers. The model typically falters in the "same class" categories. For e.g. between one govt. issued card vs the other like shown below sample UID card

sample vehicle reg card

as you can see both look different to humans but kinda have the same dense text in the center of the image. There's false positives across both these categories.

What additional features can we use along with just the ones extracted from a pre-trained model ? (we understand ImageNet wasn't trained on such images but the features it extracts, along with a simple LSTM+DENSE model gives us an F score of ~80%). We can try them all and use PCA to ascertain the best mix.

Here's what we tried (along with ImageNet features)

  • single dimension histogram (both color and grayscale images)

  • multi dimensional histogram (3 vectors that we stacked) this just drives the overall accuracy lower.

Most documents we read online pertain to certain domains where domain specific features can be extracted. Not sure what we do here .. please guide

  • $\begingroup$ Do you need to classify the documents in two categories or extract text from it? $\endgroup$ Commented Feb 1, 2020 at 0:47
  • $\begingroup$ Hi Shubham , both ..but as u can see each document will have different key value pairs that we will need to read to confirm. Also we will have different pre processing steps for different types of documents and a bunch of rules. BUT it all begins with a high confidence on type of document $\endgroup$ Commented Feb 1, 2020 at 7:18

1 Answer 1


One option is to frame it as object detection task, in contrast to object classification task like ImageNet.

The objects could be: text box, image, number, or QR code. Once the objects are detected, there could be series of hand-coded rules that categorize the document type.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.