we are trying to build a model to classify different types of documents as the first step in our pipeline (final goal is to read all the text). Currently we use ImageNet to extract the features and then pass it through a custom model which includes an LSTM layer and a couple of FC layers. The model typically falters in the "same class" categories. For e.g. between one govt. issued card vs the other like shown below
as you can see both look different to humans but kinda have the same dense text in the center of the image. There's false positives across both these categories.
What additional features can we use along with just the ones extracted from a pre-trained model ? (we understand ImageNet wasn't trained on such images but the features it extracts, along with a simple LSTM+DENSE model gives us an F score of ~80%). We can try them all and use PCA to ascertain the best mix.
Here's what we tried (along with ImageNet features)
single dimension histogram (both color and grayscale images)
multi dimensional histogram (3 vectors that we stacked) this just drives the overall accuracy lower.
Most documents we read online pertain to certain domains where domain specific features can be extracted. Not sure what we do here .. please guide