5
From the Amazing Blog - FloydHub Blog- Attention Mechanisms
Attention Mechanisms
Attention takes two sentences, turns them into a matrix where the words of one sentence form the columns, and the words of another sentence form the rows, and then it makes matches, identifying relevant context. This is very useful in machine translation.
When we think about ...
4
Don't get hung up on the word "govern" here. $W_{ax}$, $W_{ay}$ and $W_{aa}$ are simply the weights and they play in principle the same role weights play in feed forward network (except that feedforward networks do not have $W_{aa}$):
$W_{ax}$ are the weights from your input layer to the first hidden layer (just as they are in feedforward networks)
$W_{ay}$ ...
2
My question is, when we calculate partial derivative with respect to one parameter (e.g. weight between input x1 and 1st hidden layer neuron) then we are treating all other weights and biases as constants and we are evaluating how will cost function change if we were to take a step in the direction that is represented by that particular weight. Is this ...
2
The main problem I see here is that OHE is almost never a good idea with that many categories. With neural networks you will usually get better performance by using embeddings. So instead of
X1 -{OHE}-> 10,000 -> {..} -> 1,000 you could go straight to X1 -{embedding}-> 50, where the embedding dimension should probably a lot lower than 1,000.
...
1
o(t) is not the result of concatenation of h(t-1) and x(t), but a simple matrix multiplication.
See wikipedia for further details:
https://en.wikipedia.org/wiki/Long_short-term_memory
1
I'm not sure that I understand every part of the process but there is one clear issue with it: because the CV is applied in the inner loop, there is a serious risk of overfitting the model with respect to the other parameters (feature subset, model type, sampling technique). Depending on the goal, this is not necessarily wrong but it's important to interpret ...
1
One thing you can do, is forcing your labels (${-1,1}$) to be ${0,1}$ using this simple linear transformation:
\begin{equation*}\hat{y} = (y + 1) / 2\end{equation*}
This way, -1 maps to 0, and 1 maps to 1.
For practical purposes, you can either change the outputs and labels of your model directly (before applying the original BCE), or slightly change to ...
Only top voted, non community-wiki answers of a minimum length are eligible
