Questions tagged [softmax]
The softmax tag has no usage guidance.
42
questions
0
votes
1answer
19 views
Multiple targets in a classification problem
I have a vector of length $n \gt 4$ which has exactly 4 targets, so for example [0, 0, 0, 1, 0, 1, 0, 1, 1]. I would like to know how I can modify the softmax ...
1
vote
1answer
44 views
Difference in performance Sigmoid vs. Softmax
For the same Binary Image Classification task, if in the final layer I use 1 node with Sigmoid activation function and ...
0
votes
2answers
16 views
Attention transformation - matrices
Could somebody explain which matrix dimension should be found here - K? and if it is for example 3X3, should I use just 9?
0
votes
0answers
9 views
Matrix dimensions issue on calculating Softmax Derivatives
I am trying to get the matrix dimensions right for computing derivative of a two layers network where the last layer is softmax function.
For simplicity I am only interested to get derivatives of W2 w....
0
votes
2answers
33 views
In classification task, is it possible that a truly classified data has a higher loss compared to a miscallisfied data?
Given a classifier using softmax, is it possible that say, for data point a which our model has correctly classified, has higher loss compared to data point ...
0
votes
0answers
11 views
A low-degree polynomial alternative for softmax?
I want to train on a low footprint NN which gives decent accuracy on IoTs. I find that the classification using softmax is tedious and very slow. I am looking for a low-degree polynomial alternative ...
1
vote
0answers
15 views
1
vote
1answer
39 views
Derivative of a custom loss function with the logistic function
I have costum loss function with $\mu ,p, o, u, v$ as variables and $\sigma$ is the logistic function.
I need to derive this loss function.
Due to multiple variables in the loss function, I need to ...
2
votes
0answers
32 views
Using 2 nodes in the output sigmoid activation function for 2 mutually exclusive classes is somehow giving good results than softmax
I know for two mutually exclusive classes softmax is the best activation function in the output layer. However, somehow (2, softmax) and even (1,sigmoid) are giving average results and (2, sigmoid) as ...
1
vote
0answers
10 views
Ensemble of different reservoirs (echo state networks)
Suppose I want to do reservoir computing to classify the input to the proper category (e.g. recognizing a handwritten letter).
Ideally, after training a single reservoir and testing it, there would be ...
3
votes
3answers
347 views
Dot product for similarity in word to vector computation in NLP
In NLP while computing word to vector we try to maximize log(P(o|c)). Where P(o|c) is probability that o is outside word, given that c is center word.
Uo is word vector for outside word
Vc is word ...
0
votes
0answers
17 views
Dependent Classifiction
Most of the classification problems I have seen assume that the classes are kind of independent. For example, a digit classification uses a softmax output that penalizes the output prediction error ...
0
votes
0answers
13 views
In multinomial logistic regression, how to explain the softmax outputs properly?
I tried to solve multiclass problem ("cat", "dog", "horse") problem and figured out that the more words in test text, the more difference between classes.
I grouped the ...
0
votes
1answer
100 views
Softmax regression cost function code [closed]
I really do not understand what does this code do
M = sparse.coo_matrix(([1]*n, (Y, range(n))), shape=(k,n)).toarray()
The code is related to calculating the ...
2
votes
0answers
324 views
Precision-Recall Curve Intuition for Multi-Class Classification Utilizing SoftMax Activation [closed]
I am running a CNN image multi-class classification model with Keras/Tensorflow and have established about a 90% overall accuracy with my best model trial. I have 10 unique classes I am trying to ...
1
vote
0answers
109 views
Valid approach? LogSoftmax during training, Softmax during inference
I am training a classifier assigning one of four possible classes to each frame in a preprocessed audio stream using pytorch. I am using cross-entropy loss as the loss function for training. It is ...
1
vote
2answers
69 views
Lower loss always better for Probabilistic loss functions?
I am working on an neural net int Tensorflow that predicts percentages for win, draw, loss for given data of a game. The labels I provide are always {1, 0, 0}, {0, 1, 0} or {0, 0, 1}. After some ...
1
vote
3answers
44 views
What if multiple final prediction values for multi-class Neural Network are equal
If, for example, your final prediction for a multi-class problem, say for ["mouse","cat","dog","lion"], is [0.1,0.3,0.3,0.3], should the neural network predict that this data is "cat","dog" or "lion"? ...
1
vote
0answers
175 views
Generalized softmax derivative for implementation with any loss function
I am currently taking some deep learning and neural network (NN) courses, and in addition to performing the course work, am implementing my own "toolkit" of NN techniques to better my understanding of ...
0
votes
1answer
25 views
When can you reorder log operations?
For example, you can reorder a softmax + nl (negative likelihood)
to log_softmax + nll (negative log-likelihood)
Essentially changing log(softmax(x)) to softmax(log(x))
However, what are the ...
2
votes
1answer
62 views
Problem with chain rule in softmax layer when differentiated separately
I have some problems with backpropagation in softmax output layer. I know how it should work but if I try to apply the chain rule in the classical way, I get different results compared to when Softmax ...
1
vote
0answers
17 views
Can someone please explain Lovasz softmax loss? as its a bit difficult to understand why it works well from the original paper [duplicate]
Lovasz Softmax is used a lot these days for segmentation problem and the original paper is really bad at explaining why it works.
1
vote
0answers
53 views
Balance two crossentropy losses with different number of neurons
I have a model with a few outputs, each output with shape:
Shape: (batch_size, labels_1) -> softmax -> ...
0
votes
0answers
24 views
Keras Softmax is Hardmaxing for some reason
I am new to Keras and am bit confused at the moment:
...
-1
votes
1answer
1k views
Softmax gives output vector whose sum is greater than 1 in Pytorch
I am a newbie to PyTorch. I was trying out the following network architecture to train a multi-class classifier. I used Softmax at the output layer and cross entropy as the loss function. However, the ...
1
vote
0answers
54 views
Multi-label classification with missing labels
I have a neural network that generates a vector that represents the class probabilities. Since it is a multilabel classification problem, I'm supposed to train the network using sigmoid + binary cross-...
0
votes
2answers
366 views
How to calculate Temperature variable in softmax(boltzmann) exploration
Hi I am developing a reinforcement learning agent for a continous state/discrete action space. I am trying to use boltmzann/softmax exploration as action selection strategy. My action space is of size ...
1
vote
0answers
33 views
temperature variable in boltzmmann-exploration in reinforcement learning
I have been using epsilon greedy action selection strategy and recently have come across boltzmann(softmax) action selection strategy. One thing I am not clear about boltzmann exploration is the ...
1
vote
0answers
179 views
boltzmann-exploration(softmax exploration) in reinforcement learning
I have started learning reinforcement learning and as a part of it I am exploring the action selection strategies available. I am comparing epsilon-greedy vs boltzmann exploration(softmax exploration)....
19
votes
4answers
9k views
Gumbel-Softmax trick vs Softmax with temperature
From what I understand, the Gumbel-Softmax trick is a technique that enables us to sample discrete random variables, in a way that is differentiable (and therefore suited for end-to-end deep learning)....
3
votes
1answer
345 views
Softmax activation predictions not summing to 1
I am a beginner with rnns, consider this sample code
...
1
vote
1answer
667 views
Backpropagation with log likelihood cost function and softmax activation
In the online book on neural networks by Michael Nielsen, in chapter 3, he introduces a new cost function called as log-likelihood function defined as below
$$ C = -ln(a_y^L) $$
Suppose we have 10 ...
3
votes
0answers
171 views
Can I turn any binary classification algorithms into multiclass algorithms using softmax and cross-entropy loss?
Softmax + cross-entropy loss for multiclass classification is used in ML algorithms such as softmax regression and (last layer of) neural networks. I wonder if this method could turn any binary ...
3
votes
3answers
3k views
Do I need to standardize my one hot encoded labels?
I'm trying to do a simple softmax regression where I have features (2 columns) and a one hot encoded vector of labels (two categories: left = 1 and Right = 0). Do I need to standardize just the ...
0
votes
2answers
91 views
Is the Cross Entropy Loss important at all, because at Backpropagation only the Softmax probability and the one hot vector are relevant?
Is the Cross Entropy Loss (CEL) important at all, because at Backpropagation (BP) only the Softmax (SM) probability and the one hot vector are relevant?
When applying BP, the derivative of CEL is the ...
3
votes
1answer
3k views
Pytorch doing a cross entropy loss when the predictions already have probabilities
So, normally categorical cross-entropy could be applied using a cross-entropy loss function in PyTorch or by combing a logsoftmax with the negative log likelyhood function such as follows:
...
2
votes
1answer
395 views
Why use different variations of Softmax in training and validation for neural networks with Pytorch?
Specifically, I'm working on a modeling project, and I see someone else's code that looks like
...
2
votes
2answers
531 views
Why do we use a softmax activation function in Convolutional Autoencoders?
I have been working on an image segmentation project where I have created a convolutional autoencoder. I saw this image and implemented it using Keras.
At the output layer, the author has used the ...
1
vote
1answer
79 views
Softmax function result for already normalized probabilities
Isn't the aim of softmax function normalizing the probabilities such that they all sum to 1? So when we apply this method to the already normalized numbers, it would change them. what do these new ...
1
vote
1answer
35 views
Does the score output of a classification model has a global meaning?
The scores-output layer contains the class scores that the model generated for the current sample and it is passed thru the softmax layer to get the final output in the form of a probabilities vector.
...
3
votes
1answer
478 views
Multiclass Classification with Decision Trees: Why do we calculate a score and apply softmax?
I'm trying to figure out why when using decision trees for multi class classification it is common to calculate a score and apply softmax, instead of just taking the averages of the terminal nodes ...
63
votes
6answers
123k views
Cross-entropy loss explanation
Suppose I build a neural network for classification. The last layer is a dense layer with Softmax activation. I have five different classes to classify. Suppose for a single training example, the <...
