Questions tagged [loss-function]
A function used to quantify the difference between observed data and predicted values according to a model. Minimization of loss functions is a way to estimate the parameters of the model.
200
questions
1
vote
4answers
57 views
Neural net only works for small datasets
I have a neural network that attempts to solve some regression problem.
The network works fine when the dataset has a small number of training examples, lets say 20. It overfits of course, but the ...
1
vote
0answers
12 views
How to backpropogate error from convolutional layer with respect to the input when using multiple channels
I have been attempting to implement a Convolutional Neural Network in python and have run into a bit of a roadblock. When backpropogating the error in a convolutional layer let us say that we receive ...
1
vote
0answers
8 views
How does the Backpropagation through time work?
I have to write a paper on LSTMs and I want to explain why LSTMs exist in the first place. According to some papers and books because usual RNNs had problems with vanishing gradients and the LSTM has ...
0
votes
1answer
15 views
Understanding REINFORCE loss
The loss used in REINFORCE algorithm is confusing me.
From Pytorch documentation :
loss = -m.log_prob(action) * reward
We want to minimize this loss.
If a ...
0
votes
0answers
20 views
Intuitive explanation of Lovasz Softmax loss for Image Segmentation problems
Lovasz Softmax is used a lot these days for segmentation problem and the original paper is really bad at explaining why it works.
0
votes
1answer
18 views
Why doesn't the binary classification log loss formula make it explicit that natural log is being used?
I'm completing a DataCamp course where we are introduced to the log loss formula for binary classification:
Two scenarios are given to show how the formula is used. One with p=0.1 and one with p=0.5....
2
votes
1answer
23 views
How to train a neural network for high recall?
I would like to train a neural network for named entity recognition to tag an unlabeled dataset of texts. The generated labels will then be checked via a crowdsourcing platform. The goal is to ...
0
votes
0answers
16 views
Similarity of perceptron criterion and SVM
In the book "Neural Networks and Deep Learning" by Aggarwal there is an exercise 2.10.1:
Consider the following loss function for training pair $(\overline{X},y)$:
$$L=max(0, a -y(\overline{W} \...
1
vote
1answer
71 views
Poor performance of regression model for imbalanced data
I am trying to train a neural network model to solve a regression problem. The specificity of my dataset is that it has something like an exponential distribution of target values (imbalanced). ...
0
votes
0answers
19 views
Trainable parameter in a Keras loss function
all,
I am trying to implement a loss function with trainable parameters in a Keras model.
I managed it off keras.backend as follows:
...
0
votes
0answers
16 views
Rank by sorting the output probabilities of a binary classifier or by “learning to rank”?
I have a binary classification problem (5M points, only around 0.01% is positive). As output I want to give a list of data points sorted by "positiveness", and then only the top ...
0
votes
2answers
14 views
Is the Cross Entropy Loss important at all, because at Backpropagation only the Softmax probability and the one hot vector are relevant?
Is the Cross Entropy Loss (CEL) important at all, because at Backpropagation (BP) only the Softmax (SM) probability and the one hot vector are relevant?
When applying BP, the derivative of CEL is the ...
0
votes
0answers
12 views
Understanding about RNN loss
Problem
I am now taking Andrew Ng's deep learning course on Coursera. Everything is great but when it comes to RNN, I sometimes feel confused. Here is a question about RNN (or more specifically, the ...
1
vote
1answer
61 views
Pytorch doing a cross entropy loss when the predictions already have probabilities
So, normally categorical cross-entropy could be applied using a cross-entropy loss function in PyTorch or by combing a logsoftmax with the negative log likelyhood function such as follows:
...
1
vote
1answer
27 views
better confussion matrix higher LogLoss ? Is that possible>
I have tried a 2 different versions of a gbm in a multinomial classification problem. The second model results in better confusion matrix but in worse Log Loss value (at the test sample). How is that ...
0
votes
0answers
12 views
Does validation loss increase if the dataset is small?
This is my loss vs epoch image...
You can see that my model converges too early. However, the frustating point is validation loss does not decrease accordingly compared to training loss. I am doing ...
0
votes
0answers
9 views
Loss function for multi-class classifiction where output variable is a level i.e the various classes are dependent on each other
Let's say we are classifying Images of cat , fish and human. Classifying a cat as human is as wrong as classifying it as fish, so here the normal loss functions/ metrics like Confusion matrix is fine.
...
1
vote
1answer
27 views
Tweedie Loss for Keras
We are currently using XGBoost model with Tweedie loss for solving a regression problem which works very good, now I wanted to move our model to Keras and experience with neural networks, do anybody ...
3
votes
1answer
74 views
How do I create a Keras custom loss function for a one-hot-encoded binary classifier?
I have a CNN binary classifier with one-hot-encoded labels that I've written using Keras and it's just not training to the metric I want to encourage. My data is very imbalanced (91% class 0, 9% class ...
1
vote
0answers
26 views
Huge cost not converging well with TensorFlow logistic regression
I try to use Logistic Regression for a dataset which contains 15 numeric features and 4238 rows of examples. The calculated cost started at 415.91, and converged when the cost was reduced to 220.119 ...
0
votes
0answers
69 views
Can Keras automatically calculate my custom loss gradient?
I'm using a custom loss function that I wrote completely in tensorflow. I am using Tensorflow 2.0 though so I am not sure that it is all part of one graph, I don't know how that works. Also, I use the ...
0
votes
0answers
69 views
Trying to get shannon entropy of image in tensorflow 2.0
I'm trying to incorporate the shannon entropy of an image in my custom loss function in tensorflow. I tried to just find a function that does that in tensorflow and I found tfp.distributions....
0
votes
0answers
32 views
Trying to build a custom loss function in Keras/Tensorflow but get an error
I am using tensorflow 2.0 and I built a loss function that has two inputs, y_pred and y_true. I made sure to use only tensorflow operations throughout the entire function (it's long) and to return a ...
0
votes
1answer
16 views
How is the “loss” calculated which is supplied by the callback log in Keras?
I.e. categorical cross entropy? binary cross entropy? Something else?
Or is it perhaps the loss function which you pass into the model.compile method?
0
votes
0answers
18 views
Minimizing cross entropy vs minimizing negative probabilities
The cross entropy loss can be written as $L_1 = -\sum_i\sum_c y_{ic}\log P_{ic}$, where $i$ represents images and $c$ are the classes. $y_{ic}=1$ for the correct class.
Instead of L_1 I can minimize ...
1
vote
1answer
73 views
Is Categorical Crossentropy Loss always bounded in the [0, 1] interval?
Is Categorical Crossentropy always bounded between 0 and 1, or is it possible that during training of a Neural Network it can get higher values?
More specifically, I'm referring to the TensorFlow 2.0 ...
0
votes
1answer
52 views
Why such a big difference in number between training error and validation error?
Question
Why such a big difference between my 'Train loss' and 'Validation loss' as shown in the picture below? Is it a signal that my codes are wrong and my trained network is wrong as well?
Some ...
0
votes
0answers
25 views
The optimal coefficient of a linear line for minimizing the loss function
Suppose we have data points $(x^1, y^1), (x^2, y^2), ... , (x^n, y^n)$ and we want to fit a line $y = ax$ to them and set a such that it minimizes the loss function. what should the a be here to ...
1
vote
1answer
122 views
Is it a acceptable way to write a loss function in this form?
I found a loss function of a perceptron on a book is in this form
$$ L(w,b) = - \sum\limits_{x_i \in M}y_i(wx_i+b) $$
In the context of machine learning loss functions, is it acceptable to put $x_i$...
6
votes
1answer
132 views
What makes binary cross entropy a better choice for binary classification than other loss functions?
I'm reading this
post where I came across this quote "Cross-entropy is the default loss function to use for binary classification problems."
But what about it makes it the default and presumably best ...
0
votes
1answer
29 views
2
votes
0answers
19 views
Get derivatives from your NN
How can I get the gradient of a node in the NN with respect to another one? I need to train a NN, which for the sake of simplicity has 2 neurons as input (x, y), a neuron as a bottleneck (z), and 2 ...
2
votes
0answers
22 views
Negative evidence lower bound?
As my title already says: can the ELBO be negative?
$ELBO_\lambda = KL[q_\lambda(w)||P(w)] - \mathbb{E}_q[\log P(\mathcal{D|w})] $
Can I theoretically adjust my prior $P(w)$ such that $KL=0$ and ...
0
votes
1answer
70 views
Why does putting a 1/2 in front of the squared error make the math easier?
Per wiki, the mean squared error (MSE) looks like:
$$
\operatorname {MSE} ={\frac {1}{m}}\sum _{i=1}^{m}(y_{i}-{\hat y_{i}})^{2}
$$
The professor added a $1\over2$ in front of the formula and ...
1
vote
2answers
68 views
What is the relationship between “square loss” and “Mean squared error”?
Loss functions are important part of machine learning.
Square loss is one of the most popular loss functions.
Mean squared error (MSE) measures the average squared difference between the estimated ...
0
votes
0answers
9 views
Training a neural network whose loss function involves training a second neural network
I have a fully-connected feed-forward neural network whose output I'll denote by the function $u_{1}(x;\theta)$. My loss function is defined by
$$
L_{1}(x;\theta) = \max_{u_{2} \in V} f(u_{1}(x;\theta)...
1
vote
0answers
17 views
Stateful loss function and online regression
Working in tensorflow + keras I'm trying to define a custom loss function. N.B. I'm more interested in the value of the loss rather than the actual value of the predictions (this will be used for ...
0
votes
0answers
22 views
Choosing loss function in Keras for prediction binary_crossentropy or categorical_crossentropy
What loss function in keras should I chose for binary_crossentropy or categorical_crossentropy?
I have data like : $w1,w2,w3,w2,w2,w1,w3,w5,w9,w5,w4...$
I want to predict sequence:
input: $w1,w2,w3$...
0
votes
0answers
34 views
Keras: Struggle with loss-functions w/ additional parameters and merging them
I'm quite new to the field of data-science and fiddle around with some time-series-data, in my case stock-data. I want to learn a neural-net to forecast them, as everyone has tried to do some time I ...
0
votes
0answers
13 views
The principle and understanding of adversarial training
In the paper《ADVERSARIAL TRAINING METHODS FOR SEMI-SUPERVISED TEXT CLASSIFICATION》and its related papers.
The researcher apply the adversarial perturbation to word embeddings.
Why do this methods ...
1
vote
1answer
73 views
Understanding Training and Test Loss Plots
I have attached a figure that contains 6 subplots below. Each shows training and test loss over multiple epochs. Just by looking at each graph, how can I see which one is the best? Which ones are ...
3
votes
1answer
34 views
Hinge Loss understanding and proof
I hope this doesn't come off as a silly question, but I am looking at SVMs and in principle I understand how they work. The idea is to maximize the margin between different classes of point (within ...
0
votes
1answer
135 views
Optimization of a custom loss function
I want to implement a deep learning model in Keras, but I want to use my own loss function, i.e. custom loss. If I implement some loss function and use Keras Functional API for the model, do I need to ...
0
votes
1answer
35 views
Reducing Regression to Classification
If a regression problem is reduced to classification, does minimizing the classification loss translate to minimizing regression error and hence better regression performance?
1
vote
0answers
39 views
Intuition behind the loss function in Deep Q learning?
I'm currently following a tutorial but I got stuck at the deep Q learning model. According to my understanding of neural networks they predict an approximate function for the inputs given with the ...
1
vote
0answers
149 views
Best neural-network loss function for multiple output when order doesn't matter?
I have a problem where my network is to output 3 values, and they are supposed to match three target values, but I don't care about the ordering. At present, I created a loss function which computes ...
0
votes
1answer
38 views
MSE vs Cross Entropy for training with facial landmark (pose) heatmaps
I am trying to reimplement the excellent paper https://github.com/1adrianb/face-alignment-training in tensorflow. I have successfully defined the network and downloaded the LSD3D-W dataset. I am ...
1
vote
0answers
35 views
Keras deviance custom loss
I am trying to use deviance in order to optimize my network. Note that my $y_{true}$ values are equal to 0 more than in 90% of cases (can also be 1 or 2).
The deviance is calculated as : $2 \times (...
2
votes
1answer
29 views
Having trouble figuring out how loss was calculated for SQuAD task in BERT paper
The BERT Paper
https://arxiv.org/pdf/1810.04805.pdf
Section 4.2 covers the SQuAD training.
So from my understanding, there are two extra parameters trained, they are two vectors with the same ...
1
vote
0answers
29 views
Suggestion for choosing (building) loss funciton
I would like to build a supervised learning model M satisfying the following conditions:
Training data {X, Y}, where $x \in R^m$ and $y \in R^n$
Assume: ...
