0
$\begingroup$

I have a use case where I have large texts, and a lot of it. Pretty often the sequence length exceeds 1000 tokens. I need a lower dimensional compression of the texts as an input for a classifier. The classifier needs to be able to be trained on a very small amount of data. However, I would like to bring as much information as possible into the lower dimensional compression of the text.

Generally I see approaches where the AE tries to generate text to similar to the input. However, I think for long sequences must be slow, and I am not actually interested in generating understandable sentences. As long as the high dimensional input is compressed into some gist/bottleneck I am happy.

I am thinking of the following approach:

  • context encoder: encodes the text in context-aware embeddings. This can be done for example using CNNs, LSTMs, or Attention mechanisms. The input is text.
  • encoder: encodes the context-aware embeddings into a gist/bottleneck. This is similar to the encoder of a normal autoencoder.
  • decoder: decodes the gist/bottleneck, and attempts to rebuild the input of encoder. This is similar to the encoder of a normal autoencoder.
  • loss The loss function is measuring the reconstruction loss between the encoder inputs and decoder outputs.

To summarize it is a normal autoencoder-like structure, but the network does not attempt to reconstruct the right tokens. I have not seen any approach like this so far, so I am not sure if it will work.

Would this be a valid way to proceed? Any thoughts?

Improve this question
$\endgroup$
5
  • $\begingroup$ How would you measure reconstruction loss if you're not reconstructing? Also, VAEs are structurally different from AEs, I think your question is about AEs not VAEs. $\endgroup$
    – Andy
    Sep 23, 2022 at 13:33
  • $\begingroup$ why not text embeddings like bert or word2vec or variations? $\endgroup$
    – Nikos M.
    Sep 23, 2022 at 16:45
  • $\begingroup$ Yes I am sorry for the mistake. I changed it. Regarding the loss, I was thinking to take something like the (euclidean) distance between the output of the context-encoder (some vector or matrix encoding the text), and the output of the decoder having the same dimensions. $\endgroup$
    – thijsvdp
    Sep 23, 2022 at 16:53
  • $\begingroup$ @NikosM. Yes I have pretrained embeddings that encode the words. However, for the classification task(s) I have very small amount of labels available generally speaking. One solution would be to take the average of the embeddings to get a text encoding, but this often leads to too much loss of information in my case. More complex nets lead to overfitting and instability. I was thinking that by using AEs I can leverage all the documents I have available to learn to 'summarize' the documents into a high-quality, low-dimensional space. Then for the classification tasks, I can use that vector. $\endgroup$
    – thijsvdp
    Sep 23, 2022 at 17:04
  • $\begingroup$ To make it clear, I have a lot of documents. However, for each of the classification tasks I generally have very few labels. $\endgroup$
    – thijsvdp
    Sep 23, 2022 at 17:08

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.