I have a use case where I have large texts, and a lot of it. Pretty often the sequence length exceeds 1000 tokens. I need a lower dimensional compression of the texts as an input for a classifier. The classifier needs to be able to be trained on a very small amount of data. However, I would like to bring as much information as possible into the lower dimensional compression of the text.

Generally I see approaches where the AE tries to generate text to similar to the input. However, I think for long sequences must be slow, and I am not actually interested in generating understandable sentences. As long as the high dimensional input is compressed into some gist/bottleneck I am happy.

I am thinking of the following approach:

  • context encoder: encodes the text in context-aware embeddings. This can be done for example using CNNs, LSTMs, or Attention mechanisms. The input is text.
  • encoder: encodes the context-aware embeddings into a gist/bottleneck. This is similar to the encoder of a normal autoencoder.
  • decoder: decodes the gist/bottleneck, and attempts to rebuild the input of encoder. This is similar to the encoder of a normal autoencoder.
  • loss The loss function is measuring the reconstruction loss between the encoder inputs and decoder outputs.

To summarize it is a normal autoencoder-like structure, but the network does not attempt to reconstruct the right tokens. I have not seen any approach like this so far, so I am not sure if it will work.

Would this be a valid way to proceed? Any thoughts?

  • $\begingroup$ How would you measure reconstruction loss if you're not reconstructing? Also, VAEs are structurally different from AEs, I think your question is about AEs not VAEs. $\endgroup$
    – Andy
    Sep 23 at 13:33
  • $\begingroup$ why not text embeddings like bert or word2vec or variations? $\endgroup$
    – Nikos M.
    Sep 23 at 16:45
  • $\begingroup$ Yes I am sorry for the mistake. I changed it. Regarding the loss, I was thinking to take something like the (euclidean) distance between the output of the context-encoder (some vector or matrix encoding the text), and the output of the decoder having the same dimensions. $\endgroup$
    – thijsvdp
    Sep 23 at 16:53
  • $\begingroup$ @NikosM. Yes I have pretrained embeddings that encode the words. However, for the classification task(s) I have very small amount of labels available generally speaking. One solution would be to take the average of the embeddings to get a text encoding, but this often leads to too much loss of information in my case. More complex nets lead to overfitting and instability. I was thinking that by using AEs I can leverage all the documents I have available to learn to 'summarize' the documents into a high-quality, low-dimensional space. Then for the classification tasks, I can use that vector. $\endgroup$
    – thijsvdp
    Sep 23 at 17:04
  • $\begingroup$ To make it clear, I have a lot of documents. However, for each of the classification tasks I generally have very few labels. $\endgroup$
    – thijsvdp
    Sep 23 at 17:08


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy