Problem : Document classification of scanned financial documents using Text (OCR on the images) and Images. The documents are both structural (forms, tables), unstructured (letters, descriptions, paragraphs of text) and semi-structured (may have paragraphs and forms in the same page).

Research Direction : The idea is to create a multi-model pipeline which uses text data (possible models : word2vec, doc2vec or LSTM/RNN), images (possible model: CNN) and connects the models in a coherent and mutually differentiable way that, one model (say image model) tries to learn better when the other model(text model) is not able to learn and contributes towards the right classification (changing weights doesn't effect loss).

Normal(ensemble) approach adopted for multi-model training:

  1. Train both (text and image) models separately and get an output vector for each.
  2. Concatenate both vectors and train a Fully-Connected NN with a softmax layer at the end (document classes).

enter image description here

Limitation (in my case): When the FC-NN is trained, the back-propagation is only limited up-till the start of the FC-NN. Which means that the two models (text and image) are not updated (weights are not updated in both models based on the error evaluated by the loss function in the FC-NN).

Proposed Approach

  1. Have a pipeline of three models, where image and text models are connected with a FC-NN at the end. But in this case, the loss evaluated in FC-NN is back-propagated uptil the input of the image and text models. (See the following image)
  2. Image and Text models will not have a cold-start. these will be first trained separately and the weights will be used in the combined pipeline.

As and example please see the following image.

enter image description here


Backgroud This is a research idea, derived by an actual problem i have. I have tried separate models individually, also the normal multi-model approach.

I want suggestions and opinions about the plausibility of such a solution. First, i want to validate my thought, whether it makes sense and is possible. Also if you can refer a code repo which does something like that, it will be great. Moreover, if you find something wrong in my understanding or have questions, please feel free to comment.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.