1
Have a look at the weights of your model at each step and the gradients that are being applied. In many cases the gradients are of order 10^-10 or smaller, meaning that the weights of the model basically do not change at all. The reason for this is that a neural network is sensitive to the scale of the data. It is therefore often good practice to scale your ...
1
I found the answer - there is no difference.
According to the paper "Semi-Orthogonal Low-Rank Matrix Factorization for Deep Neural Networks": "Time Delay Neural Networks (TDNNs), also known as one dimensional
Convolutional Neural Networks (1-d CNNs)..."
1
They are the same, as far as i can see. The name convolution usually applies to spatial not time dimensions, but that is only convention. Wikipedia also links the two.
1
Preliminary note
as far as I know, people may use the term feature extraction in slightly different ways:
referring to automatic methods used for dimensionality reduction which involve transforming the features (and not only selecting a subset of features).
referring to the general process of designing and engineering the features before training/testing a ...
1
If the masking were only applied in the first layer, the self-attention in the subsequent layers would bring to each position information from future tokens.
Let's break it down with numbers:
At layer $i$, if causal masking is applied, the output at position $t$ contains information about layer $i-1$ at positions $1..t-1$, that is, $L_{i,t} = f_i(L_{i-1,1},....
1
I think that the issue depends on what you'd expect the model to learn:
If the model is supposed to "know" the users it has seen during training, i.e. exploit the user id in order to infer particular choices for a specific user, then I don't see the point in adding this kind of frequency feature: the model already "knows" what choices ...
Only top voted, non community-wiki answers of a minimum length are eligible
