As far as I've seen, transformer-based architectures are always trained with classification tasks (one-hot text tokens for example). Are you aware of any architectures using attention and solving regression tasks? Could one build a regressive auto-encoder for example? How would normalization fit into this (as LayerNorm destroys some of the information from the input)?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
In the simplest case, doing regression with Transformers is just a matter of changing the loss function.
BERT-like models that use the representation of the first technical token as an input to the classifier. You can replace the classifier with a regressor and pretty much nothing will change. The error from the regressor will get propagated to the rest of the network and you can both train the regressor and fine-tune/train the underlying Transformer.
Also, I don't think that layer normalization causes severe information loss. It is already there when the network is trained, so the rest of the network parameters need to take care of that, which should not be a problem because the gradients "know very well" that there was a normalization layer.
-
$\begingroup$ Regarding the information loss, I was more referring to the autoencoder scenario. Because of LayerNorm, mean and variance of each input sequence will be completely destroyed. With that in mind, how could autoencoder ever learn to replicate input? Gradients "know very well" that there was a normalization layer in terms of learned affine transform, but a portion of information is truly lost. $\endgroup$ Jun 25, 2020 at 9:50
$\begingroup$
$\endgroup$
Transformers are increasingly being used in the area of temporal forecasting, which is autoregressive by nature and thus a good fit. Lim et al. (2021) recently announced "temporal fusion transformers" in which they perform quantile regression https://www.sciencedirect.com/science/article/pii/S0169207021000637.
I have not experienced any issues with normalization, although I normalize my data before feeding it into the transformer.
