New answers tagged


As Jindřich has said, Q, K, V come from previous computations, they are not trained directly with backpropagation. However, the weights $W_i^Q, W_i^K, W_i^V$ are trained directly with backpropagation. Expanding on this, in the "Attention is all you need paper", in the self attention used by the encoder and decoder, Q, K, V are the same matrix. If ...


If I got it correctly, you might have the following scenearios: labeled samples with tag A or B per date-time index, having as informative attributes the present + some lag values of interest --> (this would be a standard classification approach without time ordering bein necessary) sliding window of samples (what you mean by subsequences) --> here ...


In principle, it is possible to reuse the special tokens as you describe. However, according to research, you should not freeze BERT, but fine-tune the whole model with your data, in order to obtain better translation quality. Another option would be to reuse just the embeddings instead of the whole model.

Top 50 recent answers are included