I am working on a document classifier that can perform the classification based on the document structure as well. My plan is to get the word embedding as well as the word coordinates and somehow combine the two features and pass it through a Graph Convolutional Network (GCN) to generate a graph embedding which I can then use to train a classifier. I was referencing this paper and they do data extraction by first getting the text embedding and image embedding and then combining them using elementwise addition. I was wondering what would be a good solution to combine both text features and positional features (x,y,height and width).

  • $\begingroup$ maybe a vectorised form of words (eg word2vec) makes the embedding easier? $\endgroup$
    – Nikos M.
    Jan 15 at 17:53

When text is available as scanned image:

  1. Divide your image into small grids.
  2. Assign each grid a row/column number like (i,j)
  3. Now to your word vector append 2 more cells which are the row and column number of cell to which the word belongs.

When text is available as Have the document in the form of html. Then have the embeddings for entire html DOM tree of that document, which would include both tags and the actual text. This way html tags would be giving the spatial/positional information.

  • $\begingroup$ The documents I'm referring to are scanned images. I then run it through an OCR engine to get the text and their coordinates in the document. My question is, what is the best feature fusion technique to combine the word embedding and postions? $\endgroup$ Jan 18 at 9:56
  • $\begingroup$ updated my answer above $\endgroup$
    – Vivek Dani
    Jan 26 at 18:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.