1

Have a look at the weights of your model at each step and the gradients that are being applied. In many cases the gradients are of order 10^-10 or smaller, meaning that the weights of the model basically do not change at all. The reason for this is that a neural network is sensitive to the scale of the data. It is therefore often good practice to scale your ...


1

I found the answer - there is no difference. According to the paper "Semi-Orthogonal Low-Rank Matrix Factorization for Deep Neural Networks": "Time Delay Neural Networks (TDNNs), also known as one dimensional Convolutional Neural Networks (1-d CNNs)..."


1

They are the same, as far as i can see. The name convolution usually applies to spatial not time dimensions, but that is only convention. Wikipedia also links the two.


1

Preliminary note as far as I know, people may use the term feature extraction in slightly different ways: referring to automatic methods used for dimensionality reduction which involve transforming the features (and not only selecting a subset of features). referring to the general process of designing and engineering the features before training/testing a ...


1

If the masking were only applied in the first layer, the self-attention in the subsequent layers would bring to each position information from future tokens. Let's break it down with numbers: At layer $i$, if causal masking is applied, the output at position $t$ contains information about layer $i-1$ at positions $1..t-1$, that is, $L_{i,t} = f_i(L_{i-1,1},....


1

I think that the issue depends on what you'd expect the model to learn: If the model is supposed to "know" the users it has seen during training, i.e. exploit the user id in order to infer particular choices for a specific user, then I don't see the point in adding this kind of frequency feature: the model already "knows" what choices ...


Only top voted, non community-wiki answers of a minimum length are eligible