I have a use case where I have large texts, and a lot of it. Pretty often the sequence length exceeds 1000 tokens. I need a lower dimensional compression of the texts as an input for a classifier. The classifier needs to be able to be trained on a very small amount of data. However, I would like to bring as much information as possible into the lower dimensional compression of the text.
Generally I see approaches where the AE tries to generate text to similar to the input. However, I think for long sequences must be slow, and I am not actually interested in generating understandable sentences. As long as the high dimensional input is compressed into some gist/bottleneck I am happy.
I am thinking of the following approach:
- context encoder: encodes the text in context-aware embeddings. This can be done for example using CNNs, LSTMs, or Attention mechanisms. The input is text.
- encoder: encodes the context-aware embeddings into a gist/bottleneck. This is similar to the encoder of a normal autoencoder.
- decoder: decodes the gist/bottleneck, and attempts to rebuild the input of encoder. This is similar to the encoder of a normal autoencoder.
- loss The loss function is measuring the reconstruction loss between the encoder inputs and decoder outputs.
To summarize it is a normal autoencoder-like structure, but the network does not attempt to reconstruct the right tokens. I have not seen any approach like this so far, so I am not sure if it will work.
Would this be a valid way to proceed? Any thoughts?