Questions tagged [loss-function]
A function used to quantify the difference between observed data and predicted values according to a model. Minimization of loss functions is a way to estimate the parameters of the model.
529
questions
0
votes
0
answers
12
views
Parametric loss function
I would like to train a network to predict two continuously-valued outputs, x and y, from 64 continuously-valued inputs.
For each set of 64 input values, there isn't a single "correct" or &...
- 101
1
vote
2
answers
38
views
Neural network not learning at all
I am training a MLP on a tabular dataset, the pendigits dataset. Problem is that training loss and accuracy are more or less stable, while validation and test loss and accuracy are completely constant....
- 221
0
votes
0
answers
47
views
Weighting and loss function for multi-dimensional output on ECG neural network in Tensorflow
I am working on a DNN that is training on ecg data with a shape of [None,1,2500] and output shape of [None,12,19] where 19 is a ...
0
votes
0
answers
12
views
Problem of constant shift in prediction for neural network regression model with gradient-domain loss function
I'm training a regression model using neural network which is trained on MSE of both output and spatial gradient of output. With some simplification, the model is:
$$
y = f(\mathbf{x};\theta)
$$
where ...
- 1
1
vote
0
answers
9
views
NANs, Infinities, and very large losses with normalizing flows
I am new to normalizing flows and have been trying to use them with a high-dimensional dataset, and I have been running into very large numbers and errors with sampling that don't occur when I use a ...
- 31
0
votes
1
answer
29
views
Why is cross entropy loss averaged and not used directly as a sum during model training(such as in neural networks)
Why is the cross entropy loss for all training examples(or the training examples in a batch) averaged over size of the training set(or batch size) ?
Why is it not just summed and used ?
0
votes
0
answers
17
views
What is log likelihood/maximum likelihood loss?
Very naively, how might one conceptualize log likelihood/maximum likelihood? What situations is it used in? Why would someone prefer this sort of loss function compared to something like an L2 or MSE ...
- 31
0
votes
0
answers
13
views
Weighting loss functions for multi task learning
I am training a multi-task neural network which is predicting a binary target variable, an 18-class target, and a 17-class target. I am calculating the cross-entropy loss for each task, then summing ...
- 1
0
votes
0
answers
24
views
How does cross-entropy loss change with the number of classes?
How does the value of the cross-entropy loss function vary with the number of classes being predicted?
Formally, if the loss function is
$$
L = - \sum_{x \in X} P^*(x) \log P(x)
$$
where $P^*(\cdot)$ ...
- 1
0
votes
0
answers
6
views
Approaching multiple records for one observation; radiomics of 2D slices of a 3D object
Background
I am trying to create a model that can predict Type 2 diabetes in a patient based on MRI scans of their thigh muscle. Previous literature has shown that fat deposition in the muscle of ...
0
votes
1
answer
10
views
Which is the loss function used for validating a CF Recommender System?
I am developing (from scratch) a memory-based CF Recommender System based on movielens dataset.
My CF RS uses a URM (User Rating Matrix) where r_ij contains the rating the user i gave to movie j (or ...
- 101
0
votes
0
answers
77
views
The val_loss is nan, but loss is printing. Both train and validation losses are nan in model.evaluate(), and the acc improves during training
There is a 2-class classification problem, and my loss function is custom. The labels are categorical, and the final activation function is Softmax. During the training, the loss is printed, but the ...
0
votes
0
answers
11
views
Ground truth as a function of weights in Keras
I have a convolutional neural network that takes an image an outputs a value between -1 and 1. If the image is an array $I$, and the network transforms the array such that $\text{output} = f(I) \in [-...
- 71
4
votes
0
answers
86
views
How Does the Reward Model in ChatGPT Calculate Losses?
Reading the InstructGPT paper(which seems to be what ChatGPT was built off of), I found this equation for the reward function.
However, I'm struggling to understand how this equation is used to ...
- 141
0
votes
0
answers
74
views
Custom loss function in keras with class weights for each batch
I am new to deep learning and tensorflow. I am working on a speech binary classification problem, trying to replicate a research paper. Number of samples in class 1 are 2700 approx and in class 2 are ...
- 1
0
votes
3
answers
169
views
Differentiable vs Non Differentiable loss function in ML
ML Question: What is a differentiable loss function and why does it matter?
For example, for a given Input training set; the loss function is: L(y,F(x)),}
Is this ...
- 310
1
vote
0
answers
92
views
problem on designing a custom_loss function
I am using CNN to solve a regression problem in a supervised manner. i have input data(X_train) and the target data(y_train). I allow the network to train and during training process in each batch of ...
- 73
0
votes
0
answers
22
views
How to calculate the gradient with triplet loss learning?
I have a CNN (convolutional neural network) that I train for face recognition. When teaching, I choose 3 images: Anchor, Positive, Negative. I pass each of them through CNN. Then I calculate the ...
0
votes
1
answer
54
views
Can I have 0 loss in the validation set and still have bad accuracy?
I am starting in the world of deep neural networks and doing a series of tests with a convolutional model, I have found the following case:
The accuracy in the training set is much better (around 0.85)...
- 302
2
votes
1
answer
46
views
How to implement linear regression
I am having difficulty achieving the same result as in sklearn while implementing linear regression model from scratch.
After adjusting the learning rate, I obtained an AUC of 0.694 for this binary ...
- 151
0
votes
0
answers
28
views
Understanding loss function/ implementing my own
I am currently working on an ETA prediction LightGBM model (regression tree) for which I want the negative residuals to be penalized higher than the positive. I understand that a custom loss function ...
0
votes
1
answer
53
views
Define a custom distance between classes in Keras
How is it possible to make a classification with custom distance between classes in Keras?
For example, let's say I need to classify betweenA1,...
- 311
0
votes
0
answers
16
views
How does backpropagation through accuracy work?
I'm using a specific constraint on my predicted logits and adding it to the loss.
In a nutshell, this constraint tries to minimize cross-overlap between the channels of my predictions. I'm using ...
- 13
2
votes
1
answer
187
views
Validation and training loss of a model are not stable
Below I have a model trained and the loss of both the training dataset (blue) and validation dataset (orange) are shown. From my understanding, the ideal case is that both validation and training loss ...
- 231
0
votes
0
answers
142
views
Custom loss function for binary classificatio in Keras gets error: No gradients provided for any variable
I have a binary classification problem. However, I don't really care about fp and fn values. What I want to achieve is that the <...
- 43
0
votes
0
answers
20
views
2 neural netowork trained differently but ending up at the same loss means both the model are identical?
I have 2 neural netowks and I am training them in different way. 1 neural network is generating good predictions and ending up with a certain loss. Another neural network is not doing good predictions....
- 101
0
votes
0
answers
16
views
About the Evaluation method of the Market 1501 ReID dataset
The market 1501 dataset has train, query and gallery folders, each containing multiple views of people from multiple cameras. I would like to understand how to evaluate a model (trained with triplet ...
- 101
0
votes
0
answers
39
views
Why backpropagation is done in every epoch when loss is always scalar?
I understand the backpropagation algorithm that it calculates the derivate of loss with respect to all the parameters in the neural network. My question is this derivate is constant right because the ...
- 101
0
votes
0
answers
7
views
What inference can we make from this deep learning model's loss plot?
what reasons caused this type of output, simple overfit and underfit concepts does not apply here right, do they?
Does local optimal points and learning rate parameter to the optimizers have an impact ...
- 11
0
votes
0
answers
15
views
General analytical form of KL loss
The form of KL loss that I am familiar with only requires you to specify mu and sigma, which means it doesn't work particularly well when the target (prior) distribution is more complex in terms of ...
- 45
0
votes
0
answers
16
views
Why we do not use CrossEntropy loss funtion in Yolo
As we know, in Yolo Object Detection model, all elements in the loss function are from squared error function. Why they do not use CrossEntropyLoss function in it?
- 253
0
votes
0
answers
9
views
Efficient range loss function
I'm looking for an efficient loss function, that instead of value would provide me with a range for the value. The function I've written for this looks like follows:
...
- 101
0
votes
0
answers
17
views
How do I find the number of local minima in a loss function?
Consider the loss function $L(A_1, ω_1, ..., A_d, ω_d)=\displaystyle\sum_{n=1} ^{m}(Y_j-\displaystyle\sum_{k=1} ^{d}A_k\sin(ω_kt_j))^2$ used to fit a time series to a sum of simple oscillations with ...
- 1
1
vote
0
answers
34
views
Calculationg perplexity (in natural language processing) manually
I am trying to understand Perplexity within Natural Language Processing as a metric more fully. And I am doing so by creating manual examples to understand all the component parts. Is the following ...
- 125
0
votes
2
answers
140
views
Training and validation loss are almost the same (perfect fit?)
I am developing an ANN from scratch which classifies MNIST digits.
These are the curves I get using only one hidden layer composed of 100 neurons activated by ...
- 101
0
votes
0
answers
36
views
Optimization on a convex function used in a loss function
I am currently creating a deep learning model which deals with classification and regression problem together such that each class has continuous value within an interval of real numbers in common ...
- 1
0
votes
1
answer
17
views
Binary crossentropy loss
When we have a binary classification problem, we use a sigmoid activation function in the output layer+ a binary crossentropy loss. We also need to one hot encode the target variable.This s a binary ...
- 221
0
votes
1
answer
145
views
How to understand a large result of torch.nn.NLLLoss() with correct predicts?
I'm learning the usage of torch.nn.NLLLoss() and torch.nn.LogSoftmax(), and I'm confused about the results of them.
For example:
...
- 101
0
votes
0
answers
16
views
KL divergence between two multivariate gaussians where $p$ is $N(\mu, I)$
We know if we try to get $D_{KL}(q||p)$, where $p$ is a standard normal distribution, so mean is 0, variance is the identity matrix, and $q$ is a multivariate normal distribution, it can be calculated ...
- 109
0
votes
0
answers
25
views
How to improve the $R^2$ score on an autoencoder model when the loss(KLDivergence) and validation MAE is giving desired scores?
I have been training an autoencoder for data with 25k feats and 1k data points. The $R^2$ score is coming negative on every epoch, and around -27 on both train and test sets, although the MAE is 0.7 ...
0
votes
1
answer
27
views
How do I know that my weights optimizer have found the best weights?
I am new to deep learning and my understanding of how optimizers work might be slightly off. Also, sorry for a third-grader quality of images.
For example if we have simple task our loss to weight ...
- 3
1
vote
1
answer
21
views
Which loss function to use for a convolution NN for noise removal of high resolution images
My task is to remove small random spots from my 4 mega pixel images.
My strategy was to feed a convolution network these images as I have the true images without the spots in them.
The current loss ...
- 11
0
votes
1
answer
35
views
Is there a way I can double the punishment when model mis-classing to a specific class?
As the title I asked.
For example: a model that predicts the probability of a stock price rising/falling. Let's say this is a triple-classification problem.
If it predicts "RISING", while ...
- 101
5
votes
2
answers
1k
views
Why using a partial derivative for the loss function?
What is the purpose of computing the partial derivative of the loss function in order to find the best parameters that minimize the error?
Considering the loss function of a linear model, we want to ...
- 149
1
vote
3
answers
102
views
How to learn steep functions using neural network?
I am trying to use a neural network to learn the below function. In total, I have 25 features and 19 outputs. The above image shows the distribution of two features with respect to one of the outputs....
- 61
1
vote
0
answers
108
views
How can I write a custom loss function to punish lower predicted values?
I am trying to write a custom loss function for XGBRegressor that needs to punish predicted values that are under some arbitrary threshold. The code I came up with does not affect the results at all, ...
- 11
7
votes
3
answers
2k
views
Is Pearson correlation a good loss function?
I want to do a data science project. I want to use price history to predict future prices.
I want to use correlation(y, y_pred) as my loss function but I found it's ...
- 73
0
votes
0
answers
75
views
Triplet + Classification loss for Person-Reid
I am trying out a Person-Reid model on a custom dataset using Triplet Loss + Classification Loss (with label smoothing) on a custom dataset.
Following are the configurations and the graphs:
1. Triplet ...
- 13
2
votes
2
answers
504
views
SGD performing better than Adam in Random minority oversampling, I don't know what is the reason. Help
So my dataset image before and after balancing looks like this:
But when I train with Adam(0.0001) and SGD(0.0001), the results are very different. Why? What is going on under the hood?
This is ...
- 21
0
votes
1
answer
12
views
What is the effect Cross-Multi-Labeling/Annotation on learning process?
I have a philosophical question regarding training convolution neuronal network. I am work on training NN for purpose of detection of Window and Window blind. This is an issue of cross labels; that is,...
