# Questions tagged [softmax]

The tag has no usage guidance.

66 questions
Filter by
Sorted by
Tagged with
35 views

### Candidate sampling

In the context of extreme multiclass classification with softmax, if I use candidate sampling where each training sample has different output node meanings, how does the network determine at inference ...
71 views

### When applying softmax over multiple dimensions of a tensor, does the order in which those dimensions are matter?

Lets say i have a tensor of order 256, dimensions indexed from 0 to 255. Lets say i am writing a function implementing the softmax operation because i am a newbie and i want to understand the ...
• 101
1 vote
106 views

### Why does scaled dot-product attention use softmax?

I am trying to understand the reasoning behind the Transformer architecture. In "Attention is all you need", the weights for the scaled dot-product attention is defined as the scaled dot-...
26 views

### Numerical issue with softmax regression implementation on MNIST

I'm having numpy numerical issues with my implementation of softmax regression/multiclass logistic regression on the MNIST dataset. The numpy exp and log numerical issue goes away when I divide the x ...
165 views

### What is the benefit of the exponential function inside softmax?

I know that softmax is: $$softmax(x) = \frac{e^{x_i}}{\sum_j^n e^{x_j}}$$ This is an $\mathbb{R}^n \implies \mathbb{R}^n$ function, and the elements of the output add up to 1. I understand that the ...
• 123
1 vote
775 views

### PyTorch CrossEntropyLoss and Log_SoftMAx + NLLLoss give different results

As per PyTorch documentation CrossEntropyLoss() is a combination of LogSoftMax() and NLLLoss() function. However, calling CrossEntropyLoss() gives different results compared to calling LogSoftMax() ...
• 113
178 views

### How does softmax work for vectors?

In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
1 vote
62 views

### I have a question about Transformer's Q, K, V

I think the cosine similarity of negative values has its own meaning. If you softmax the cosine similarity of Q and K, wouldn't it prevent Transformer from using information with the opposite meaning?
149 views

### Why does softmax perform well on MNIST but poorly on EMNIST letters?

I am learning about softmax regression using Dive into Deep Learning. I have a very basic question on why softmax performs well on one dataset and poorly on another. I tried modifying the results from ...
76 views

### error useing soft max gives outputs greater than 1

I am using Hugging Face AutoModelForSequenceClassification, model is roberta, using it for text classification. There are 3 classes. The output is: ...
581 views

### Binary classification works with softmax, but not sigmoid

I am doing a binary classification problem for seizure classification. I split the data into Training, Validation and Test with the following sizes and shapes dataset_X = ...
19 views

### Why is the optimal output out of domain in A2C?

If each state has an optimal action, then the optimal actions distribution vector is a one-hot vector kind of like [0,0,1,0,0,0]. But with algorithms like A2C, we ...
423 views

### How to understand a large result of torch.nn.NLLLoss() with correct predicts?

I'm learning the usage of torch.nn.NLLLoss() and torch.nn.LogSoftmax(), and I'm confused about the results of them. For example: ...
• 101
325 views

### What is the advantage of using Euler's number ($e^x$) instead of another base in the softmax equation?

I understand the softmax equation is $\boldsymbol{P}(y=j \mid x)=\frac{e^{x_{j}}}{\sum_{k=1}^{K} e^{x_{k}}}$ My question is: why use $e^x$ instead of say, $3^x$. I understand $e^x$ is it's own ...
• 53
50 views

### Normalizing softmax by dividing by its maximum?

Reading this paper, I'm struggling to understand the step with the question mark (page 3). The formula for $\textbf r$ uses $\textbf q_i$ (no tilde), but the numeric values in the following paragraph ...
• 103
13 views

### What configuration of output neurons to use for detecting bias

I am trying to make a deep learning model that detects political bias in media articles for my local community. There are two political parties here and I have a dataset of biased articles from both. ...
• 125
1 vote
33 views

1 vote
190 views

### Using 2 nodes in the output sigmoid activation function for 2 mutually exclusive classes is somehow giving good results than softmax

I know for two mutually exclusive classes softmax is the best activation function in the output layer. However, somehow (2, softmax) and even (1,sigmoid) are giving average results and (2, sigmoid) as ...
• 19
1 vote
24 views

### Ensemble of different reservoirs (echo state networks)

Suppose I want to do reservoir computing to classify the input to the proper category (e.g. recognizing a handwritten letter). Ideally, after training a single reservoir and testing it, there would be ...
• 111
4k views

### Dot product for similarity in word to vector computation in NLP

In NLP while computing word to vector we try to maximize log(P(o|c)). Where P(o|c) is probability that o is outside word, given that c is center word. Uo is word vector for outside word Vc is word ...
• 129
529 views

### Softmax regression cost function code [closed]

I really do not understand what does this code do M = sparse.coo_matrix(([1]*n, (Y, range(n))), shape=(k,n)).toarray() The code is related to calculating the ...
1k views

### Precision-Recall Curve Intuition for Multi-Class Classification Utilizing SoftMax Activation [closed]

I am running a CNN image multi-class classification model with Keras/Tensorflow and have established about a 90% overall accuracy with my best model trial. I have 10 unique classes I am trying to ...
• 159
1 vote
328 views

### Valid approach? LogSoftmax during training, Softmax during inference

I am training a classifier assigning one of four possible classes to each frame in a preprocessed audio stream using pytorch. I am using cross-entropy loss as the loss function for training. It is ...
• 13
1 vote
369 views

### Lower loss always better for Probabilistic loss functions?

I am working on an neural net int Tensorflow that predicts percentages for win, draw, loss for given data of a game. The labels I provide are always {1, 0, 0}, {0, 1, 0} or {0, 0, 1}. After some ...
• 80
1 vote
56 views

### What if multiple final prediction values for multi-class Neural Network are equal

If, for example, your final prediction for a multi-class problem, say for ["mouse","cat","dog","lion"], is [0.1,0.3,0.3,0.3], should the neural network predict that this data is "cat","dog" or "lion"? ...
1 vote
253 views

### Generalized softmax derivative for implementation with any loss function

I am currently taking some deep learning and neural network (NN) courses, and in addition to performing the course work, am implementing my own "toolkit" of NN techniques to better my understanding of ...
31 views

### When can you reorder log operations?

For example, you can reorder a softmax + nl (negative likelihood) to log_softmax + nll (negative log-likelihood) Essentially changing log(softmax(x)) to softmax(log(x)) However, what are the ...
• 199
80 views

### Problem with chain rule in softmax layer when differentiated separately

I have some problems with backpropagation in softmax output layer. I know how it should work but if I try to apply the chain rule in the classical way, I get different results compared to when Softmax ...
• 153
1 vote
29 views

### Can someone please explain Lovasz softmax loss? as its a bit difficult to understand why it works well from the original paper [duplicate]

Lovasz Softmax is used a lot these days for segmentation problem and the original paper is really bad at explaining why it works.
1 vote
112 views

### Balance two crossentropy losses with different number of neurons

I have a model with a few outputs, each output with shape: Shape: (batch_size, labels_1) -> softmax -> ...
45 views

### Keras Softmax is Hardmaxing for some reason

I am new to Keras and am bit confused at the moment: ...
• 285