Skip to main content

Questions tagged [softmax]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

Implementing softmax regression using numpy

I was trying to implement softmax regression but my model failed why? ...
samsamradas's user avatar
0 votes
1 answer
31 views

When applying softmax over multiple dimensions of a tensor, does the order in which those dimensions are matter?

Lets say i have a tensor of order 256, dimensions indexed from 0 to 255. Lets say i am writing a function implementing the softmax operation because i am a newbie and i want to understand the ...
Maciej's user avatar
  • 101
0 votes
0 answers
45 views

Why does scaled dot-product attention use softmax?

I am trying to understand the reasoning behind the Transformer architecture. In "Attention is all you need", the weights for the scaled dot-product attention is defined as the scaled dot-...
Reinis Mazeiks's user avatar
0 votes
0 answers
22 views

Numerical issue with softmax regression implementation on MNIST

I'm having numpy numerical issues with my implementation of softmax regression/multiclass logistic regression on the MNIST dataset. The numpy exp and log numerical issue goes away when I divide the x ...
KaizerBox's user avatar
0 votes
0 answers
13 views

Using auxiliary softmaxes to measure impact of each submodule on the final softmax classifier

I am attempting to assess the impact of various submodules (CNN 1D, CNN 2D, CNN 3D, FFNN) on the final classifier of the neural network that i am currently building. The neural network itself is ...
glaand's user avatar
  • 101
2 votes
1 answer
131 views

What is the benefit of the exponential function inside softmax?

I know that softmax is: $$ softmax(x) = \frac{e^{x_i}}{\sum_j^n e^{x_j}}$$ This is an $\mathbb{R}^n \implies \mathbb{R}^n$ function, and the elements of the output add up to 1. I understand that the ...
Victor2748's user avatar
0 votes
0 answers
20 views

Dealing with noise in softmax output

I have a device with an accelerometer and gyroscope (6-axis). The device sends live raw telemetry data to the model 40 samples for each input, 6 values per sample (accelerometer xyz, gyroscope xyz). ...
Sterling Duchess's user avatar
0 votes
0 answers
34 views

Why does softmax give probability of 1 for outlier class?

I am trying to do open set classification by using a temperature scaled softmax and then interpreting the output probabilities as a confidence metric. However, for complete outlier inputs, the ...
Tommy Wolfheart's user avatar
1 vote
1 answer
597 views

PyTorch CrossEntropyLoss and Log_SoftMAx + NLLLoss give different results

As per PyTorch documentation CrossEntropyLoss() is a combination of LogSoftMax() and NLLLoss() function. However, calling CrossEntropyLoss() gives different results compared to calling LogSoftMax() ...
cbelwal's user avatar
  • 113
0 votes
1 answer
131 views

How does softmax work for vectors?

In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
Ruediger's user avatar
1 vote
1 answer
52 views

I have a question about Transformer's Q, K, V

I think the cosine similarity of negative values has its own meaning. If you softmax the cosine similarity of Q and K, wouldn't it prevent Transformer from using information with the opposite meaning?
 chord sketcher's user avatar
0 votes
1 answer
122 views

Why does softmax perform well on MNIST but poorly on EMNIST letters?

I am learning about softmax regression using Dive into Deep Learning. I have a very basic question on why softmax performs well on one dataset and poorly on another. I tried modifying the results from ...
user676464327's user avatar
0 votes
1 answer
64 views

error useing soft max gives outputs greater than 1

I am using Hugging Face AutoModelForSequenceClassification, model is roberta, using it for text classification. There are 3 classes. The output is: ...
prajwal rao's user avatar
0 votes
0 answers
552 views

Binary classification works with softmax, but not sigmoid

I am doing a binary classification problem for seizure classification. I split the data into Training, Validation and Test with the following sizes and shapes dataset_X = ...
Mohammed Nafie's user avatar
0 votes
1 answer
18 views

Why is the optimal output out of domain in A2C?

If each state has an optimal action, then the optimal actions distribution vector is a one-hot vector kind of like [0,0,1,0,0,0]. But with algorithms like A2C, we ...
RyuGood0's user avatar
0 votes
1 answer
357 views

How to understand a large result of torch.nn.NLLLoss() with correct predicts?

I'm learning the usage of torch.nn.NLLLoss() and torch.nn.LogSoftmax(), and I'm confused about the results of them. For example: ...
EvilRoach's user avatar
  • 101
5 votes
1 answer
267 views

What is the advantage of using Euler's number ($e^x$) instead of another base in the softmax equation?

I understand the softmax equation is $\boldsymbol{P}(y=j \mid x)=\frac{e^{x_{j}}}{\sum_{k=1}^{K} e^{x_{k}}}$ My question is: why use $e^x$ instead of say, $3^x$. I understand $e^x$ is it's own ...
Codedorf's user avatar
0 votes
1 answer
43 views

Normalizing softmax by dividing by its maximum?

Reading this paper, I'm struggling to understand the step with the question mark (page 3). The formula for $\textbf r$ uses $\textbf q_i$ (no tilde), but the numeric values in the following paragraph ...
Tim's user avatar
  • 103
0 votes
0 answers
13 views

What configuration of output neurons to use for detecting bias

I am trying to make a deep learning model that detects political bias in media articles for my local community. There are two political parties here and I have a dataset of biased articles from both. ...
Dani's user avatar
  • 125
1 vote
0 answers
33 views

Deriving a binary logistic classifier from a multi class logistic classifier

Given a multi class logisitic classifier $f(x)=argmax(softmax(Ax + \beta))$, and a specific class of interest $y$, is it possible to construct a binary logistic classifier $g(x)=(\sigma(\alpha^T x + b)...
user1767774's user avatar
0 votes
1 answer
3k views

neural network binary classification softmax logsofmax and loss function

I am building a binary classification where the class I want to predict is present only <2% of times. I am using pytorch The last layer could be logosftmax or <...
user2543622's user avatar
2 votes
1 answer
520 views

How to prove Softmax Numerical Stability?

I was playing around with the softmax function and tried around with the numerical stability of softmax. If we increase the exponent in the numerator and denominator with the same value, the output of ...
Nicoinlas's user avatar
0 votes
1 answer
403 views

Can a single label be a vector/matrix in a neural network and not a scalar?

My training data consists of individual sentences and each sentence has a few labels (say 10) and each of these labels has a discrete score from 1-10 -- so in essence, a single training example has a ...
zyezye11's user avatar
0 votes
1 answer
24 views

Train a model when input can contain a smaller options output with the correct output

I have service order lines to charge customers, each line needs to be set to an actual product. If the customer had only one product, so all lines are set to that product. But, if the there are many ...
AJ AJ's user avatar
  • 111
0 votes
0 answers
153 views

Is there a Softmax-like transformation with scale-invariance and linarity?

At the moment I'm using XGBoost to generate a prediction of probabilities with a custom objective-function to build something like an expert system. To do so I need to transform the raw XGBoost ...
Someone2's user avatar
  • 111
1 vote
0 answers
34 views

What meaning does exp function have? [closed]

Is there any problem that solution(or algorithm) would be exp function? Let's say f(x)=2^x. It's an answer of a problem when you would like to know how many pieces would be made when you fold a paper ...
sugu's user avatar
  • 21
1 vote
1 answer
515 views

Should I apply Softmax before calculating metrics Precision or similar?

I am using PyTorch Lightning (there is no tag for this and I don't have enough reputation to create one) and am facing a multi classification problem. My loss function is ...
c0mr4t's user avatar
  • 173
2 votes
1 answer
245 views

Keras: Custom output layer for multiple multi-class classifications

Hello, I’m quite new to machine learning and I want to build my first custom layer in Keras, using Python. I want to use a dataset of 103 dimensions to do classification task. The last fully connected ...
meyer_mit_ai's user avatar
1 vote
0 answers
37 views

Is the Cross entropy cost function the same as the Cross entropy loss?

Is the Cross entropy cost function defined as $J(\Theta) = -\frac{1}{m}\sum_{i=1}^{m}\sum_{k=1}^{K}y_{k}^{(i)}log(\hat{p}_{k}^{(i)})$ the same as the one implemented in ...
rober_dinero's user avatar
0 votes
1 answer
119 views

"Up or down but not sideways" bimodal time series prediction - what is the best way to model it?

Say I have a time series (e.g. bitcoin price). I want to predict tomorrow's price, specifically tomorrow's % change in price from today. Let's say this is gaussian distributed, with the mean at 0%. If ...
Cortexelus's user avatar
0 votes
0 answers
157 views

Softmax Derivative

I've been trying to build a neural network from scratch in python over the last few weeks. I've gotten it working with sigmoid, but trying to implement softmax has been killing me, entirely thanks to ...
Grant Allan's user avatar
0 votes
1 answer
73 views

Multiple targets in a classification problem

I have a vector of length $n \gt 4$ which has exactly 4 targets, so for example [0, 0, 0, 1, 0, 1, 0, 1, 1]. I would like to know how I can modify the softmax ...
Dennis's user avatar
  • 1
3 votes
1 answer
345 views

Difference in performance Sigmoid vs. Softmax

For the same Binary Image Classification task, if in the final layer I use 1 node with Sigmoid activation function and ...
Eric Cartman's user avatar
0 votes
2 answers
83 views

Attention transformation - matrices

Could somebody explain which matrix dimension should be found here - K? and if it is for example 3X3, should I use just 9?
Adolf Miszka's user avatar
0 votes
2 answers
42 views

In classification task, is it possible that a truly classified data has a higher loss compared to a miscallisfied data?

Given a classifier using softmax, is it possible that say, for data point a which our model has correctly classified, has higher loss compared to data point ...
m0ss's user avatar
  • 103
1 vote
0 answers
32 views

Convnet with peculiar loss function not learning!

Im using this loss function: ...
Cosapocha's user avatar
1 vote
1 answer
149 views

Derivative of a custom loss function with the logistic function

I have costum loss function with $\mu ,p, o, u, v$ as variables and $\sigma$ is the logistic function. I need to derive this loss function. Due to multiple variables in the loss function, I need to ...
Stav Yosef's user avatar
1 vote
0 answers
172 views

Using 2 nodes in the output sigmoid activation function for 2 mutually exclusive classes is somehow giving good results than softmax

I know for two mutually exclusive classes softmax is the best activation function in the output layer. However, somehow (2, softmax) and even (1,sigmoid) are giving average results and (2, sigmoid) as ...
Tag_DL's user avatar
  • 19
1 vote
0 answers
24 views

Ensemble of different reservoirs (echo state networks)

Suppose I want to do reservoir computing to classify the input to the proper category (e.g. recognizing a handwritten letter). Ideally, after training a single reservoir and testing it, there would be ...
Wouter's user avatar
  • 111
3 votes
3 answers
4k views

Dot product for similarity in word to vector computation in NLP

In NLP while computing word to vector we try to maximize log(P(o|c)). Where P(o|c) is probability that o is outside word, given that c is center word. Uo is word vector for outside word Vc is word ...
Vivek Dani's user avatar
0 votes
1 answer
449 views

Softmax regression cost function code [closed]

I really do not understand what does this code do M = sparse.coo_matrix(([1]*n, (Y, range(n))), shape=(k,n)).toarray() The code is related to calculating the ...
Mostafa Atallah's user avatar
2 votes
0 answers
1k views

Precision-Recall Curve Intuition for Multi-Class Classification Utilizing SoftMax Activation [closed]

I am running a CNN image multi-class classification model with Keras/Tensorflow and have established about a 90% overall accuracy with my best model trial. I have 10 unique classes I am trying to ...
Coldchain9's user avatar
1 vote
0 answers
310 views

Valid approach? LogSoftmax during training, Softmax during inference

I am training a classifier assigning one of four possible classes to each frame in a preprocessed audio stream using pytorch. I am using cross-entropy loss as the loss function for training. It is ...
MoRoBe's user avatar
  • 13
1 vote
2 answers
310 views

Lower loss always better for Probabilistic loss functions?

I am working on an neural net int Tensorflow that predicts percentages for win, draw, loss for given data of a game. The labels I provide are always {1, 0, 0}, {0, 1, 0} or {0, 0, 1}. After some ...
Evator's user avatar
  • 70
1 vote
3 answers
56 views

What if multiple final prediction values for multi-class Neural Network are equal

If, for example, your final prediction for a multi-class problem, say for ["mouse","cat","dog","lion"], is [0.1,0.3,0.3,0.3], should the neural network predict that this data is "cat","dog" or "lion"? ...
Conor Kirby's user avatar
1 vote
0 answers
249 views

Generalized softmax derivative for implementation with any loss function

I am currently taking some deep learning and neural network (NN) courses, and in addition to performing the course work, am implementing my own "toolkit" of NN techniques to better my understanding of ...
Krothagon's user avatar
0 votes
1 answer
30 views

When can you reorder log operations?

For example, you can reorder a softmax + nl (negative likelihood) to log_softmax + nll (negative log-likelihood) Essentially changing log(softmax(x)) to softmax(log(x)) However, what are the ...
katiex7's user avatar
  • 199
2 votes
1 answer
79 views

Problem with chain rule in softmax layer when differentiated separately

I have some problems with backpropagation in softmax output layer. I know how it should work but if I try to apply the chain rule in the classical way, I get different results compared to when Softmax ...
Display name's user avatar
1 vote
0 answers
28 views

Can someone please explain Lovasz softmax loss? as its a bit difficult to understand why it works well from the original paper [duplicate]

Lovasz Softmax is used a lot these days for segmentation problem and the original paper is really bad at explaining why it works.
Prativa Das's user avatar
1 vote
0 answers
110 views

Balance two crossentropy losses with different number of neurons

I have a model with a few outputs, each output with shape: Shape: (batch_size, labels_1) -> softmax -> ...
Daniel Möller's user avatar