# Tag Info

90

The cross entropy formula takes in two distributions, $p(x)$, the true distribution, and $q(x)$, the estimated distribution, defined over the discrete variable $x$ and is given by $$H(p,q) = -\sum_{\forall x} p(x) \log(q(x))$$ For a neural network, the calculation is independent of the following: What kind of layer was used. What kind of activation was ...

13

The answer from Neil is correct. However I think its important to point out that while the loss does not depend on the distribution between the incorrect classes (only the distribution between the correct class and the rest), the gradient of this loss function does effect the incorrect classes differently depending on how wrong they are. So when you use ...

11

Let's say you have two states, $X_1$ and $X_2$, and you have a model, $M$, that produces a score $M(X_i)$ for each state (i.e, the logits). Next you can use the logits to compute some distribution $$P = softmax(\{M(X_1), M(X_2)\})$$ and take the state with the highest probability $$X=argmax_{X_i}(P)$$ But what if you actually want to sample from $P$ ...

9

Let's start with understanding entropy in information theory: Suppose you want to communicate a string of alphabets "aaaaaaaa". You could easily do that as 8*"a". Now take another string "jteikfqa". Is there a compressed way of communicating this string? There isn't is there. We can say that the entropy of the 2nd string is more as, to communicate it, we ...

4

NO, you do not standardize labels The purpose of standardization is to bring features with disparate ranges into a standard range. When the data is not standardized, features with large numerical values will tend to have a larger influence (weight) than those that are smaller numerically. Consider the Automobile Data Set (https://archive.ics.uci.edu/ml/...

4

You can implement categorical cross entropy pretty easily yourself. It is calculated as $$\text{cross-entropy} = -\frac{1}{n} \sum_{i=0}^{n} \sum_{j=0}^m \mathbf{y}_{ij} \log \hat{\mathbf{y}}_{ij}$$ where $n$ is the number of samples in your batch, $m$ is the number of classes, $\mathbf{y}_i$ is the one-hot target for example $i$, $\mathbf{\hat{y}}_i$ ...

3

From a practical and theoretical perspective, when is it beneficial to incorporate Gumbel noise into a neural network, as opposed to just using Softmax with temperature? You don't necessarily need Gumbel-Softmax to obtain "one-hot like" vectors, or the ability to differentiate through an indexing mechanism. The LSTM architecture and derived variants ...

2

Let's see how the gradient of the loss behaves... We have the cross-entropy as a loss function, which is given by $$H(p,q) = -\sum_{i=1}^n p(x_i) \log(q(x_i)) = -(p(x_1)\log(q(x_1)) + \ldots + p(x_n)\log(q(x_n))$$ Going from here.. we would like to know the derivative with respect to some $x_i$:  \frac{\partial}{\partial x_i} H(p,q) = -\frac{\partial}{...

2

Summing to 1 is just one property of the softmax function. The softmax function takes the exponential of each value and divides it by the sum of the exponentials of all values. This tends to cluster values towards the mean, as you've seen in your example. While the outputs of a softmax look and smell like probabilities, their values don't actually ...

2

For the softmax function, no matter what is the temperature, it is not the exact one-hot vector. If you could accept a soft version, it is good. However, if you choose the argmax to be the one, it is non-differentiable. One alternative way to back-propagate the gradients is by using the Straight Through Estimator (STE) trick, and directly back-propagate ...

2

$d_k$ is the dimensionality of the query/key/value vectors. In your example, the length of those vectors is 3, so $d_k = 3$

2

Assuming cross-entropy loss (as you specified in the comment), it can happens. The reason is that the loss in cross-entropy is "activated" only for the probability of the ground truth (t_i is 1 for the ground truth and 0 otherwise). Imagine a classification task with five classes, with the two following probability situations: (a) 0.4   0.5   0....

1

From what I can tell, there isn't a "right" answer to the title question. Most people I know wouldn't bother. (Indeed, one often-used scaler puts the data into the range $[0,1]$ anyway.) https://stats.stackexchange.com/questions/290929/standardizing-dummy-variables-for-variable-importance-in-glmnet https://stats.stackexchange.com/questions/359015/ridge-...

1

SoftMax is not a LOSS Function. If this is binary classification task - then you are probably using binary cross entropy as the objective / loss function. So - if the correct label is 1, and you predict 0.7, then the loss is: -1 * log(0.7) = 0.35 But if you had made a worse prediction - say 0.1, then the loss is: -1 * log(0.3) = 2.3 (higher of the two)

1

It's a loss function applied to a regression with l2 penalty on the parameters. The first square brackets can be interpreted in the following way: $- \frac{1}{n}$ has the minus because it wants to minimize. $\sum_{i=1}^{n}$ means for each data point. $\sum_{j=0}^{k-1}$ means for each class. $y_i == j$ means that the fraction after this term is calculated ...

1

I'd say it's not very common to have absolutely equal probabilities with a model that has been trained for sufficient epochs and that does not overfit hard. That said, if you happen to have equal probabilities, the choice of the class to predict should depend on the domain you're approaching, e.g: A possible implementation is like in numpy's argmax, by ...

1

In general, you cannot reorder like that. The example you give is a very special case that only works because softmax is based on the exponential function, which is the inverse function of the natural logarithm.

1

I've been also looking for the answer of this question, and I give my different view of Gumbel softmax just because I think this is a good question. From a general point of view: We use softmax normally because we need a so-called score, or a distribution $\pi_1 .. \pi_n$ for representing n probabilities of categorical variable with size n; We use Gumbel-...

1

Softmax doesnt work as an LSTM activation. You have to add a dense layer using a softmax activation after the LSTM layer. I wouls suggest using another activation in the LSTM like relu.

1

It's more of a pytorch implementation thing. log_softmax() outputs the raw logits and they are passed to NLL Loss in training. During inference you just need the probabilities so softmax will suffice. You don't use different algorithms for training and testing, the question is quite misleading as currently stated. You are using different implementations of ...

1

Since its a multi-class classification problem, each class will have its own probability value between 0 and 1 and sum of all probabilities of classes will equal to 1.

1

The output of the final layer, before softmax activation, is the log odds of the classes. Softmax isn't a linear function, so [10,20,30] going into softmax won't give the same answer as [1,2,3].

1

I would like to add a couple of dimensions to the above answers: true label = [1 0 0 0 0] predicted = [0.1 0.5 0.1 0.1 0.2] cross-entropy(CE) boils down to taking the log of the lone +ve prediction. So CE = -ln(0.1) which is = 2.3. This means that the -ve predictions dont have a role to play in calculating CE. This is by intention. On a rare occasion, it ...

Only top voted, non community-wiki answers of a minimum length are eligible