Questions tagged [loss-function]
A function used to quantify the difference between observed data and predicted values according to a model. Minimization of loss functions is a way to estimate the parameters of the model.
0
votes
0answers
16 views
Minimizing cross entropy vs minimizing negative probabilities
The cross entropy loss can be written as $L_1 = -\sum_i\sum_c y_{ic}\log P_{ic}$, where $i$ represents images and $c$ are the classes. $y_{ic}=1$ for the correct class.
Instead of L_1 I can minimize ...
1
vote
1answer
28 views
Is Categorical Crossentropy Loss always bounded in the [0, 1] interval?
Is Categorical Crossentropy always bounded between 0 and 1, or is it possible that during training of a Neural Network it can get higher values?
More specifically, I'm referring to the TensorFlow 2.0 ...
0
votes
1answer
27 views
Why such a big difference in number between training error and validation error?
Question
Why such a big difference between my 'Train loss' and 'Validation loss' as shown in the picture below? Is it a signal that my codes are wrong and my trained network is wrong as well?
Some ...
0
votes
0answers
19 views
The optimal coefficient of a linear line for minimizing the loss function
Suppose we have data points $(x^1, y^1), (x^2, y^2), ... , (x^n, y^n)$ and we want to fit a line $y = ax$ to them and set a such that it minimizes the loss function. what should the a be here to ...
1
vote
1answer
111 views
Is it a acceptable way to write a loss function in this form?
I found a loss function of a perceptron on a book is in this form
$$ L(w,b) = - \sum\limits_{x_i \in M}y_i(wx_i+b) $$
In the context of machine learning loss functions, is it acceptable to put $x_i$...
5
votes
1answer
57 views
What makes binary cross entropy a better choice for binary classification than other loss functions?
I'm reading this
post where I came across this quote "Cross-entropy is the default loss function to use for binary classification problems."
But what about it makes it the default and presumably best ...
0
votes
1answer
23 views
2
votes
0answers
19 views
Get derivatives from your NN
How can I get the gradient of a node in the NN with respect to another one? I need to train a NN, which for the sake of simplicity has 2 neurons as input (x, y), a neuron as a bottleneck (z), and 2 ...
2
votes
0answers
16 views
Negative evidence lower bound?
As my title already says: can the ELBO be negative?
$ELBO_\lambda = KL[q_\lambda(w)||P(w)] - \mathbb{E}_q[\log P(\mathcal{D|w})] $
Can I theoretically adjust my prior $P(w)$ such that $KL=0$ and ...
0
votes
1answer
61 views
Why does putting a 1/2 in front of the squared error make the math easier?
Per wiki, the mean squared error (MSE) looks like:
$$
\operatorname {MSE} ={\frac {1}{m}}\sum _{i=1}^{m}(y_{i}-{\hat y_{i}})^{2}
$$
The professor added a $1\over2$ in front of the formula and ...
1
vote
2answers
55 views
What is the relationship between “square loss” and “Mean squared error”?
Loss functions are important part of machine learning.
Square loss is one of the most popular loss functions.
Mean squared error (MSE) measures the average squared difference between the estimated ...
0
votes
0answers
8 views
Training a neural network whose loss function involves training a second neural network
I have a fully-connected feed-forward neural network whose output I'll denote by the function $u_{1}(x;\theta)$. My loss function is defined by
$$
L_{1}(x;\theta) = \max_{u_{2} \in V} f(u_{1}(x;\theta)...
1
vote
0answers
15 views
Stateful loss function and online regression
Working in tensorflow + keras I'm trying to define a custom loss function. N.B. I'm more interested in the value of the loss rather than the actual value of the predictions (this will be used for ...
0
votes
0answers
19 views
Choosing loss function in Keras for prediction binary_crossentropy or categorical_crossentropy
What loss function in keras should I chose for binary_crossentropy or categorical_crossentropy?
I have data like : $w1,w2,w3,w2,w2,w1,w3,w5,w9,w5,w4...$
I want to predict sequence:
input: $w1,w2,w3$...
0
votes
0answers
30 views
Keras: Struggle with loss-functions w/ additional parameters and merging them
I'm quite new to the field of data-science and fiddle around with some time-series-data, in my case stock-data. I want to learn a neural-net to forecast them, as everyone has tried to do some time I ...
0
votes
0answers
11 views
The principle and understanding of adversarial training
In the paper《ADVERSARIAL TRAINING METHODS FOR SEMI-SUPERVISED TEXT CLASSIFICATION》and its related papers.
The researcher apply the adversarial perturbation to word embeddings.
Why do this methods ...
1
vote
1answer
37 views
Understanding Training and Test Loss Plots
I have attached a figure that contains 6 subplots below. Each shows training and test loss over multiple epochs. Just by looking at each graph, how can I see which one is the best? Which ones are ...
3
votes
1answer
27 views
Hinge Loss understanding and proof
I hope this doesn't come off as a silly question, but I am looking at SVMs and in principle I understand how they work. The idea is to maximize the margin between different classes of point (within ...
0
votes
0answers
47 views
Optimization of a custom loss function
I want to implement deep learning model in Keras, but I want to use my own loss function, i.e. custom loss. If I implement some loss function and use Keras Functional API for the model, do I need to ...
0
votes
1answer
34 views
Reducing Regression to Classification
If a regression problem is reduced to classification, does minimizing the classification loss translate to minimizing regression error and hence better regression performance?
1
vote
0answers
32 views
Intuition behind the loss function in Deep Q learning?
I'm currently following a tutorial but I got stuck at the deep Q learning model. According to my understanding of neural networks they predict an approximate function for the inputs given with the ...
0
votes
0answers
25 views
Best neural-network loss function for multiple output when order doesn't matter?
I have a problem where my network is to output 3 values, and they are supposed to match three target values, but I don't care about the ordering. At present, I created a loss function which computes ...
0
votes
0answers
23 views
MSE vs Cross Entropy for training with facial landmark (pose) heatmaps
I am trying to reimplement the excellent paper https://github.com/1adrianb/face-alignment-training in tensorflow. I have successfully defined the network and downloaded the LSD3D-W dataset. I am ...
1
vote
0answers
30 views
Keras deviance custom loss
I am trying to use deviance in order to optimize my network. Note that my $y_{true}$ values are equal to 0 more than in 90% of cases (can also be 1 or 2).
The deviance is calculated as : $2 \times (...
2
votes
1answer
23 views
Having trouble figuring out how loss was calculated for SQuAD task in BERT paper
The BERT Paper
https://arxiv.org/pdf/1810.04805.pdf
Section 4.2 covers the SQuAD training.
So from my understanding, there are two extra parameters trained, they are two vectors with the same ...
1
vote
0answers
28 views
Suggestion for choosing (building) loss funciton
I would like to build a supervised learning model M satisfying the following conditions:
Training data {X, Y}, where $x \in R^m$ and $y \in R^n$
Assume: ...
1
vote
2answers
450 views
loss/val_loss decrease but acc/val_acc are consistent
I don't know why I am getting such good results.
...
1
vote
0answers
37 views
Is there a metric for counting AND classification simultaneously?
I'm working on a project that mixes object detection and crowd counting.
The metric for object detection is mAP, which combines the regression of the bounding boxes with the precision of the ...
1
vote
2answers
129 views
loss/val_loss are decreasing but accuracies are the same in LSTM!
I am trying to train a LSTM model, but the problem is that the loss and val_loss are decreasing from 12 and 5 to less than 0.01, but the training set ...
2
votes
1answer
60 views
Square Root Regularization and High Loss
I am testing out square root regularization (explained ahead) in a pytorch implementation of a neural network. Square root regularization, henceforth l1/2, is just like l2 regularization, but instead ...
1
vote
1answer
249 views
Pytorch : Loss function for binary classification
Fairly newbie to Pytorch & neural nets world.Below is a code snippet from a binary classification being done using a simple 3 layer network :
...
1
vote
0answers
58 views
0
votes
1answer
34 views
How can I access to loss value in Keras LSTM implementation?
I use Keras library and it's LSTM model. When I train my network I can see loss value in my program execution console. I like to know how can I access to this value ...
1
vote
0answers
66 views
Variational AutoEncoder giving negative loss
I'm learning about variational autoencoders and I've implemented a simple example in keras, model summary below. I've copied the loss function from one of Francois Chollet's blog posts and I'm getting ...
1
vote
1answer
122 views
Effects of L2 loss and smooth L1 loss
Can any one tell me what the effects of $L_2$ loss and smooth $L_1$ loss (i.e. Huber loss with $\alpha = 1$) are, and when to use each of them ?
1
vote
1answer
51 views
Why margin-based ranking loss is reversed in these two papers?
For knwoledge graph completion, it is very common to use margin-based ranking loss
In the paper:margin-based ranking loss is defined as
$$ \min \sum_{(h,l,t)\in S} \sum_{(h',l,t')\in S'}[\gamma + d(...
2
votes
0answers
275 views
Custom Lambda layer Keras outputs predictions. I get 'An operation has `None` for gradient' error
I have a Lambda layer that takes input from previous layer, makes some preprocessing. Output of the Lambda layer is a prediction, and keras.losses.mean_squared_error is used.
...
2
votes
1answer
70 views
How to recognise when to stop training based on Overfitting/Underfitting?
I am trying to train a LSTM network, over a total of 200 epochs, with hidden layer size of 100 and 1 dense layer after the LSTM layer. I have used a batch size of 10 for the same. Basically, I am ...
1
vote
1answer
33 views
Meaning of this notion in 0-1 loss?
I am reading a paper and encountered this notion:
$$1_{\{Y=1\}}$$
To me it seems to be the expression as below, but I am not entirely sure and I don't think the author explictly explained it:
<...
0
votes
1answer
23 views
Why increasing the number of units or layers does not increase the accuracy and decrease the loss? [closed]
I have an LSTM neural network; when I increase the number of units, layers, epochs or add dropout, it seems it has no effect and still I have persistent errors and accuracies like the following:
...
0
votes
0answers
187 views
Model loss and validation loss not decreasing? How to speed?
Every image was resized to 256x256 pixels. Batch size = 4. (GPU GTX 1050 memory ~4GB).
The mask R-CNN model was initial-ized using pretrained weights from COCO dataset
TRAIN_ROIS_PER_IMAGE = 8
...
2
votes
1answer
58 views
MSE loss different in Keras and PyToch
My problem is that in PyTorch I cannot reproduce the MSE loss that I have achieved in Keras.
I have trained the following model in Keras:
...
1
vote
1answer
177 views
Loss is decreasing but val_loss not! [duplicate]
If loss is decreasing but val_loss not, what is the problem and how can I fix it?
I get such vague result:
0
votes
2answers
23 views
What is purpose of partial derivatives in loss calculation (linear regression)?
I am studying ML and data science stuff from scratch. As a part of the course, I am studying how the models are derived. And for most of them, starting with the simplest - linear regression, we take ...
3
votes
1answer
65 views
How to utilize user feedback due to miss-classification when correct class label is unknown?
Suppose we are developing an app which is supposed to predict a dog's breed by it's picture. We trained a classifier (in my case an MLP) using some dataset and shipped the app to users. Now suppose ...
2
votes
2answers
121 views
Why do people use CrossEntropyLoss and not just a softmax probability as the loss?
I don't understand why one would add additional complexity to log, probabilities for the loss function of a classification Neural Network. What benefit does that have, as opposed to just using the 0-1....
1
vote
1answer
14 views
What does it mean: “Everything looks OK but loss won't decreases!”
I have written a LSTM network. It seems all the things are OK but when I train the network, I get the same loss amount about 4.9e-4 for every iterations! What is the problem? Why my network can't ...
0
votes
1answer
26 views
Difference between “reducing batch_size” and “increasing epochs” to decrease loss amount?
In my experience, both reducing batch_size and increasing epochs can decrease loss amount. But I like to know is there any ...
1
vote
1answer
50 views
Is there any standard or normal range for the amount of LSTM loss function?
I am working on a LSTM network that I get loss amounts around 4.7 e-4 . It seems adding more layers and increasing epochs don't help to decreasing it. I also using a ...
2
votes
0answers
41 views
policy gradient loss [closed]
I am confused with the process for calculating loss. My code is below:
...
