Questions tagged [softmax]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

What configuration of output neurons to use for detecting bias

I am trying to make a deep learning model that detects political bias in media articles for my local community. There are two political parties here and I have a dataset of biased articles from both. ...
user avatar
  • 125
0 votes
0 answers
5 views

How to calculate the expression for the gradient of softmax + cross entropy with respect to weights?

I'm learning cs231n on my own. The Softmax classifier has the following loss function: to make this clear: $L_i$ is the loss for a particular training input $f_j$ is the $j$th element of the vector, ...
user avatar
  • 115
0 votes
0 answers
6 views

Sample variance matrix normal distribution in R

I'm trying to perform a multinomial logistic regression in R employing the Metropolis-Hastings algorithm, considering a Matrix Normal Distribution as proposal. I'm using the function rmatrixnorm() ...
user avatar
1 vote
0 answers
17 views

Deriving a binary logistic classifier from a multi class logistic classifier

Given a multi class logisitic classifier $f(x)=argmax(softmax(Ax + \beta))$, and a specific class of interest $y$, is it possible to construct a binary logistic classifier $g(x)=(\sigma(\alpha^T x + b)...
user avatar
0 votes
0 answers
8 views

CNN Eliminate Wrong Results

I extracted images of human faces from the videos, but the model also recorded images without faces. I wrote CNN for emotion classification. In the obvious pictures, the probability is closer to a ...
user avatar
0 votes
0 answers
12 views

using logsumexp in softmax

I saw this equation in somebody's code which is an alternative approach to implementing the softmax in order to avoid underflow by division by large numbers. softmax = e^(matrix - logaddexp(matrix)) = ...
user avatar
  • 309
0 votes
1 answer
169 views

neural network binary classification softmax logsofmax and loss function

I am building a binary classification where the class I want to predict is present only <2% of times. I am using pytorch The last layer could be logosftmax or <...
user avatar
0 votes
0 answers
33 views

classification using LogSoftmax vs Softmax and calculating precision-recall curve?

In case of binary classification we could get final output using LogSoftmax or Softmax. In case of softmax we get results that add up to 1. I understand that LogSoftmax penalizes more for a wrong ...
user avatar
0 votes
0 answers
35 views

Distilling the knowledge of a binary cross entropy with sigmoid function model to a softmax model

I have a complex CNN architecture that uses a binary cross-entropy and sigmoid function for classification. However, due to hardware restraints I would like to compress my model using knowledge ...
user avatar
0 votes
0 answers
109 views

Using SVM as final layer in Convolutional Neural Network

I am working on the implementation of a hybrid CNN-SVM, where I define the use of SVM in the last layer of CNN as shown in this code: ...
user avatar
  • 95
1 vote
1 answer
70 views

How to prove Softmax Numerical Stability?

I was playing around with the softmax function and tried around with the numerical stability of softmax. If we increase the exponent in the numerator and denominator with the same value, the output of ...
user avatar
0 votes
1 answer
70 views

Can a single label be a vector/matrix in a neural network and not a scalar?

My training data consists of individual sentences and each sentence has a few labels (say 10) and each of these labels has a discrete score from 1-10 -- so in essence, a single training example has a ...
user avatar
0 votes
1 answer
20 views

Train a model when input can contain a smaller options output with the correct output

I have service order lines to charge customers, each line needs to be set to an actual product. If the customer had only one product, so all lines are set to that product. But, if the there are many ...
user avatar
  • 11
0 votes
0 answers
63 views

Is there a Softmax-like transformation with scale-invariance and linarity?

At the moment I'm using XGBoost to generate a prediction of probabilities with a custom objective-function to build something like an expert system. To do so I need to transform the raw XGBoost ...
user avatar
  • 111
1 vote
0 answers
24 views

What meaning does exp function have? [closed]

Is there any problem that solution(or algorithm) would be exp function? Let's say f(x)=2^x. It's an answer of a problem when you would like to know how many pieces would be made when you fold a paper ...
user avatar
  • 21
1 vote
1 answer
41 views

Should I apply Softmax before calculating metrics Precision or similar?

I am using PyTorch Lightning (there is no tag for this and I don't have enough reputation to create one) and am facing a multi classification problem. My loss function is ...
user avatar
  • 63
0 votes
0 answers
10 views

Approximation of a confidence scores from a neural network with a final softmax layer: Softmax vs other normalization methods

Say that there is a neural network for classification and the 2nd to last layer are 3 nodes, and the final layer is a softmax layer. During training the softmax layer is needed, but for inference it ...
user avatar
0 votes
0 answers
64 views

How to compute gradient of Softmax in backward pass of a neural network?

I am trying to implement the backward pass of a Softmax layer. As an input to my backward-function, I receive the gradient from the next upper layer and I have to ...
user avatar
2 votes
1 answer
79 views

Keras: Custom output layer for multiple multi-class classifications

Hello, I’m quite new to machine learning and I want to build my first custom layer in Keras, using Python. I want to use a dataset of 103 dimensions to do classification task. The last fully connected ...
user avatar
0 votes
0 answers
19 views

Update function for NN with logistic and sofmax

Can anyone help me confirm my work or find resources on how to come up with the update function for each layer in the Neural Network for multi-class classification problem i.e I am using logistic as ...
user avatar
  • 1
1 vote
0 answers
22 views

Is the Cross entropy cost function the same as the Cross entropy loss?

Is the Cross entropy cost function defined as $J(\Theta) = -\frac{1}{m}\sum_{i=1}^{m}\sum_{k=1}^{K}y_{k}^{(i)}log(\hat{p}_{k}^{(i)})$ the same as the one implemented in ...
user avatar
0 votes
1 answer
63 views

"Up or down but not sideways" bimodal time series prediction - what is the best way to model it?

Say I have a time series (e.g. bitcoin price). I want to predict tomorrow's price, specifically tomorrow's % change in price from today. Let's say this is gaussian distributed, with the mean at 0%. If ...
user avatar
0 votes
0 answers
41 views

Softmax Derivative

I've been trying to build a neural network from scratch in python over the last few weeks. I've gotten it working with sigmoid, but trying to implement softmax has been killing me, entirely thanks to ...
user avatar
0 votes
0 answers
26 views

Andrew Ng Deep Learning Gradient Descent of Softmax is just y_hat - y?

At about 8:30 in the video here: https://www.youtube.com/watch?v=ueO_Ph0Pyqk so for the given example with 4 classes and first ground truth y being [0,1,0,0] and y_hat being [0.3,0.2,0.1,0.4] for ...
user avatar
0 votes
1 answer
35 views

Multiple targets in a classification problem

I have a vector of length $n \gt 4$ which has exactly 4 targets, so for example [0, 0, 0, 1, 0, 1, 0, 1, 1]. I would like to know how I can modify the softmax ...
user avatar
  • 1
1 vote
1 answer
169 views

Difference in performance Sigmoid vs. Softmax

For the same Binary Image Classification task, if in the final layer I use 1 node with Sigmoid activation function and ...
user avatar
0 votes
2 answers
30 views

Attention transformation - matrices

Could somebody explain which matrix dimension should be found here - K? and if it is for example 3X3, should I use just 9?
user avatar
0 votes
2 answers
36 views

In classification task, is it possible that a truly classified data has a higher loss compared to a miscallisfied data?

Given a classifier using softmax, is it possible that say, for data point a which our model has correctly classified, has higher loss compared to data point ...
user avatar
  • 103
1 vote
0 answers
21 views

Convnet with peculiar loss function not learning!

Im using this loss function: ...
user avatar
1 vote
1 answer
72 views

Derivative of a custom loss function with the logistic function

I have costum loss function with $\mu ,p, o, u, v$ as variables and $\sigma$ is the logistic function. I need to derive this loss function. Due to multiple variables in the loss function, I need to ...
user avatar
1 vote
0 answers
53 views

Using 2 nodes in the output sigmoid activation function for 2 mutually exclusive classes is somehow giving good results than softmax

I know for two mutually exclusive classes softmax is the best activation function in the output layer. However, somehow (2, softmax) and even (1,sigmoid) are giving average results and (2, sigmoid) as ...
user avatar
  • 19
1 vote
0 answers
20 views

Ensemble of different reservoirs (echo state networks)

Suppose I want to do reservoir computing to classify the input to the proper category (e.g. recognizing a handwritten letter). Ideally, after training a single reservoir and testing it, there would be ...
user avatar
  • 111
3 votes
3 answers
1k views

Dot product for similarity in word to vector computation in NLP

In NLP while computing word to vector we try to maximize log(P(o|c)). Where P(o|c) is probability that o is outside word, given that c is center word. Uo is word vector for outside word Vc is word ...
user avatar
0 votes
1 answer
202 views

Softmax regression cost function code [closed]

I really do not understand what does this code do M = sparse.coo_matrix(([1]*n, (Y, range(n))), shape=(k,n)).toarray() The code is related to calculating the ...
user avatar
2 votes
0 answers
705 views

Precision-Recall Curve Intuition for Multi-Class Classification Utilizing SoftMax Activation [closed]

I am running a CNN image multi-class classification model with Keras/Tensorflow and have established about a 90% overall accuracy with my best model trial. I have 10 unique classes I am trying to ...
user avatar
1 vote
0 answers
169 views

Valid approach? LogSoftmax during training, Softmax during inference

I am training a classifier assigning one of four possible classes to each frame in a preprocessed audio stream using pytorch. I am using cross-entropy loss as the loss function for training. It is ...
user avatar
  • 13
1 vote
2 answers
130 views

Lower loss always better for Probabilistic loss functions?

I am working on an neural net int Tensorflow that predicts percentages for win, draw, loss for given data of a game. The labels I provide are always {1, 0, 0}, {0, 1, 0} or {0, 0, 1}. After some ...
user avatar
  • 70
1 vote
3 answers
47 views

What if multiple final prediction values for multi-class Neural Network are equal

If, for example, your final prediction for a multi-class problem, say for ["mouse","cat","dog","lion"], is [0.1,0.3,0.3,0.3], should the neural network predict that this data is "cat","dog" or "lion"? ...
user avatar
1 vote
0 answers
210 views

Generalized softmax derivative for implementation with any loss function

I am currently taking some deep learning and neural network (NN) courses, and in addition to performing the course work, am implementing my own "toolkit" of NN techniques to better my understanding of ...
user avatar
0 votes
1 answer
28 views

When can you reorder log operations?

For example, you can reorder a softmax + nl (negative likelihood) to log_softmax + nll (negative log-likelihood) Essentially changing log(softmax(x)) to softmax(log(x)) However, what are the ...
user avatar
  • 199
2 votes
1 answer
70 views

Problem with chain rule in softmax layer when differentiated separately

I have some problems with backpropagation in softmax output layer. I know how it should work but if I try to apply the chain rule in the classical way, I get different results compared to when Softmax ...
user avatar
1 vote
0 answers
22 views

Can someone please explain Lovasz softmax loss? as its a bit difficult to understand why it works well from the original paper [duplicate]

Lovasz Softmax is used a lot these days for segmentation problem and the original paper is really bad at explaining why it works.
user avatar
1 vote
0 answers
80 views

Balance two crossentropy losses with different number of neurons

I have a model with a few outputs, each output with shape: Shape: (batch_size, labels_1) -> softmax -> ...
user avatar
0 votes
0 answers
27 views

Keras Softmax is Hardmaxing for some reason

I am new to Keras and am bit confused at the moment: ...
user avatar
  • 277
-1 votes
1 answer
2k views

Softmax gives output vector whose sum is greater than 1 in Pytorch

I am a newbie to PyTorch. I was trying out the following network architecture to train a multi-class classifier. I used Softmax at the output layer and cross entropy as the loss function. However, the ...
user avatar
1 vote
0 answers
93 views

Multi-label classification with missing labels

I have a neural network that generates a vector that represents the class probabilities. Since it is a multilabel classification problem, I'm supposed to train the network using sigmoid + binary cross-...
user avatar
1 vote
2 answers
539 views

How to calculate Temperature variable in softmax(boltzmann) exploration

Hi I am developing a reinforcement learning agent for a continous state/discrete action space. I am trying to use boltmzann/softmax exploration as action selection strategy. My action space is of size ...
user avatar
  • 515
1 vote
0 answers
48 views

temperature variable in boltzmmann-exploration in reinforcement learning

I have been using epsilon greedy action selection strategy and recently have come across boltzmann(softmax) action selection strategy. One thing I am not clear about boltzmann exploration is the ...
user avatar
  • 515
1 vote
0 answers
253 views

boltzmann-exploration(softmax exploration) in reinforcement learning

I have started learning reinforcement learning and as a part of it I am exploring the action selection strategies available. I am comparing epsilon-greedy vs boltzmann exploration(softmax exploration)....
user avatar
  • 515
25 votes
4 answers
12k views

Gumbel-Softmax trick vs Softmax with temperature

From what I understand, the Gumbel-Softmax trick is a technique that enables us to sample discrete random variables, in a way that is differentiable (and therefore suited for end-to-end deep learning)....
user avatar
  • 351