Questions tagged [loss-function]
A function used to quantify the difference between observed data and predicted values according to a model. Minimization of loss functions is a way to estimate the parameters of the model.
539
questions
0
votes
0
answers
4
views
What would a good loss function to penalize small diffrences?
Using MSE loss, the model is able to learn and make predictions, but I want to improve its performance.
From epoch 1, I am already getting a very small loss because the model is minimizing relatively ...
- 1
0
votes
0
answers
14
views
How to make model know that LOW price always lesser or equal than CLOSE price?
I have this timeseries data, it retrieved based on last 1 day with 15m per timestep:
I perform global min-max scaling for ...
0
votes
0
answers
14
views
Force a single sonnected set for segmentation model using U-Net
I have a simple U-Net model for 2 classes, binary, image segmentation.
The classes are background and an single object.
The object is a connected set.
Namely the mask is a single blob of pixels with ...
- 66
0
votes
1
answer
32
views
Gradient output through custom loss function
I’m very new to Pytorch (and ML in general), so I’m having difficulty understanding what is going on WRT a custom loss/cost function I’m looking at. I understand what’s going on in the function, but I ...
0
votes
0
answers
47
views
why YOLO models multiple the loss by batch size in detection head?
here
return loss.sum() * batch_size, loss.detach() # loss(box, cls, dfl)
This line is from yolov8, but I saw similar thing in v5 too.
So far I only see this kind ...
- 101
2
votes
0
answers
23
views
Stuck with constant loss and network not learning
I am trying to predict certain function coefficients (output: a, b) based on its curve (input: frequency_response) with the help of https://github.com/Blealtan/efficient-kan (Kolmogorov-Arnold Network)...
- 151
0
votes
0
answers
9
views
How to handle sequences with crossEntropyLoss
fist of all i am ne wto the whole thing, so sorry if this is superdumb.
I'm currently training a Transformer model for a sequence classification task using CrossEntropyLoss. My input tensor has the ...
- 101
1
vote
1
answer
48
views
Is it appropriate to use KL Divergence as a loss function for a 1x3 regression model?
I have a regression model with a 1x3 output, which means it predicts three continuous values. I'm wondering if it would be appropriate to use the Kullback-Leibler (KL) Divergence as the loss function ...
- 175
2
votes
2
answers
68
views
Is there a max absolute error loss function?
I'm fitting a small model and need to use a custom loss function. I want to avoid large errors. I thought of a max absolute error function
$$ \text{MaxAE} = \max(|y-y_{pred}|) \, ,$$
with y the ...
- 123
2
votes
0
answers
46
views
Transformer model conditional probability distribution of sub-sentences
I have a simple transformer model (decoder only) which is trained on some dataset containing sentences to do next-word prediction. The model captures a probability distribution $P_{\theta}(\mathbf{a})$...
- 21
6
votes
2
answers
689
views
How do regression loss functions like MAE and MSE work although they remove the plus/minus sign?
I have a question about regression loss functions like Mean Absolute Error (MAE) and Mean Squared Error (MSE) used in deep learning.
When we calculate these losses, we remove the plus/minus sign from ...
- 175
1
vote
1
answer
80
views
Can you use the Euclidean Distance as a loss function?
While building an auto-encoder that preserves distances, i accidentally used the euclidean norm as the loss for the difference between the x and z distances that im trying to minimize. (I hope you can ...
- 11
0
votes
0
answers
27
views
Wrote a PINN to try to simulate 1-D heat transfer through a medium with constant diffusivity. The model doesn't even learn the boundary conditions!
I divide my training into 2 parts. a) Training on the boundary conditions within the domain. b) Training on the pde_loss.
The problem goes something like this: Both the ends of a 1 dimensional ...
0
votes
0
answers
15
views
Why does the TensorFlow docs use a different GAN generator loss?
As per the original paper that introduced GANs, the generator loss is given as:
$$
L_{G} = L _{BCE}(\mathbf{\vec 0}, \mathbf{D}(\mathbf{G}(\mathbf{\vec z}))) = \log(1 - \mathbf{D}(\mathbf{G}(\mathbf{\...
1
vote
2
answers
122
views
Mean Absolute Error from Scratch in NumPy
I recently tried implementing MAE from scratch in NumPy. The loss value and the slope seem to be equivalent to what Scikit-learn outputs, but for some reason the intercept value seems to converge to ...
- 11
0
votes
0
answers
9
views
How to implement CP tensor completion with extra calculations?
I am new to tensor decomposition. I want to know from a practical point of view, how to use an already known tool/library to compute CP factorization for tensor completion. Specifically, I want to ...
- 1
0
votes
0
answers
21
views
Autoencoders failing to recreate MNIST numbers
I have been having trouble trying to get a working (non-variational) autoencoder to reproduce images from the MNIST dataset. The two biggest issues is an averaging of the samples to yield a single ...
- 1
1
vote
1
answer
186
views
How to choose a loss function and how to calculate the loss for Text Generation in Generative AI?
For the classification problems, what loss functions can I choose ?
For the translation problem how do I decide whether the translation is good and how to choose a loss function?
And what about the ...
- 133
2
votes
1
answer
60
views
Custom loss function in python
I am trying to implement a custom loss function inspired by https://arxiv.org/pdf/2305.10464.pdf. That is:
$ L(\mathbf{x}) = (1-y) \left\lVert \mathbf{x_{true} - \mathbf{x_{pred}}} \right\rVert^2 + y \...
- 21
1
vote
0
answers
62
views
Multiclass matrix loss function in scikit-learn / xgboost / lightgbm
I have data with 4 classes: $c_1, c_2, c_3, c_4$.
I'd like to create a classifier which has different scaling for the loss function per class combination:
$$ \begin{bmatrix} 0 & l \left( \hat{c}_{...
- 66
0
votes
0
answers
11
views
Problem when merge two probability distribution [Pointer Network]
I'm trying to re-implement Pointer Gen Net from this paper. Ya but you don't really need to read the paper.
To sum up briefly,
I have a vector probability distribution over vocabulary called p_vocab. ...
- 101
1
vote
1
answer
54
views
Does using different optimizer change the loss landscape
I plot the landscape using this code, and I notice the landscape shape has changed a lot. My understanding is that the optimizer does not change the loss landscape. But now I'm confused if its just ...
- 127
0
votes
0
answers
15
views
How to combine a classificiation dataset with a pair-wise comparison dataset
Let's say I'm trying to train a neural network that predicts a single output [0.0, 1.0] value that correlates to photo realism which I can use either in a classification setting or for ranking. I have ...
- 201
1
vote
1
answer
42
views
How to comment on goodness of loss functions?
I have two loss functions $\mathcal{L}_1$ and $\mathcal{L}_2$ to train my model. The model is predominantly a classification model. Both $\mathcal{L}_1$ and $\mathcal{L}_2$ takes are two variants of ...
- 123
0
votes
0
answers
8
views
Add a custom loss in a public github repository on distillation
I’m trying to implement a custom loss in a public repository regarding knowledge distillation. The link to the repository is the following:
" https://github.com/DefangChen/SimKD "
The main ...
- 1
0
votes
1
answer
39
views
My custom neural network is converging but keras model not
in most cases it is probably the other way round but...
I have implemented a basic MLP neural network structure with backpropagation. My data is just a shifted quadratic function with 100 samples. I ...
1
vote
1
answer
63
views
Use of Gradient with respect to feature instead of model parameters
Generally, for any machine learning/deep learning system, we compute a loss, $L = l(x, \theta, y)$ which is a function of the input feature vector $x$ (after activation), model parameters $\theta$ (...
- 61
0
votes
0
answers
22
views
Custom loss function for collinearity of 3 embeddings
I am trying to implement a loss function that takes as input 3 embeddings and output a value that is proportional to the collinearity of the embeddings.
This is to shape the latent space of a ...
- 1
0
votes
0
answers
10
views
Cross-Entropy Loss for Prediction Distribution
I'm trying to adapt an example from the Neural Tangents Cookbook which uses a variant of Mean Squared Error (MSE) loss suited for Regression prediction distributions given by Mean and Variance as ...
- 1
0
votes
0
answers
12
views
How to split the data for speaker verification using AAM Softmax loss?
There are several models for speaker verificaiton task (wavlm-ecapa / xvector / ...).
Some of those model where trained with AAM Softmax loss, which gets the number of labels as input.
When training ...
- 413
1
vote
0
answers
20
views
How to force my model heads to learn different things?
I have an Seq2Seq model that has 2 generative LM heads. I want the two heads to focus on different features/styles while decoding. The approach that I was thinking of is adding a distance cost to the ...
0
votes
0
answers
103
views
Custom loss function for multi label classification in catboost?
I have a data frame which I want to use for multi class classification problem. There are total 6 classes (say a, b, c, d, e, f). I want to improve the precision for three classes (say a, b, c) i.e. ...
- 111
0
votes
0
answers
42
views
Setting derivative to 0 when minimizing expected loss over possible data samples
Suppose, given $x\in \mathbb{R}^d$, you want $\theta$, which is the solution of
$$
\text{argmin}_{\theta} \mathbb{E}_a[L(\theta;x,a)]
$$
where $a \sim \mathcal{N}(x,\Sigma)$ and $L$ measures a (convex)...
1
vote
0
answers
40
views
Analysis of relationship between accuracy and total loss (or cost) during training with logistic loss function and threshold 0.5
I'm trying to understanding the relationship between training accuracy and training loss in classification tasks, specifically using logistic regression. When using logistic loss as the loss function ...
- 155
0
votes
1
answer
73
views
Is my model overfitting based on my accuracy/loss curves?
Do those results indicate that my model is overfitting?
- 17
0
votes
0
answers
129
views
Generator loss not decreasing while training GAN
I’ve been attempting to create a basic GAN to generate images using this database of flowers (https://www.robots.ox.ac.uk/~vgg/data/flowers/102/).
I’ve spent a few days on this, and largely based my ...
0
votes
0
answers
67
views
Doubts on a custom loss function for regression problems
From what I read, I know we don't use log loss or cross entropy for regression problems. However, the entire logic behind binary cross entropy(say) is to firstly squeeze the y_hat between 0 and 1 (...
- 23
0
votes
0
answers
16
views
Meaning of mean squared error in multistep prediction
In multistep prediction with LSTM(keras), say we had this kind of result:
target = [[1,2,3] ,[4,5,6] ]
predictions = [[1.1,2.2,3.3] , [4.4,5.5,6.6]]
When we choose mean_squared_error as the loss ...
- 23
0
votes
0
answers
156
views
Custom Loss Function Returns Graph Execution Error: Can not squeeze dim[0], expected a dimension of 1, got 32
I have built a loss function which adds time and frequency weighted averages and variances to the MSE:
...
0
votes
0
answers
25
views
Minimize MAE loss for a target that is sum of two other targets
Working on a regression modelling task where my dataset have some feature columns, two more columns A, B and a target column T. The goal is to predict T, and minimize MAE, that is ...
0
votes
0
answers
119
views
Custom loss and metric functions including additional parameter in Keras
The following example is based on this approach.
Similar to that approach, I am wanting to pass an additional parameter with y_true for my custom metric, as both will be used in the computation of ...
- 1
0
votes
0
answers
25
views
In the GAN objective function, why do we first do we first find the D(x) that maximizes the objective function and then maximise wrt the generator?
The GAN objective function is optimised like this: argmin(argmax(L(G,D))) where the argmax finds the D (Discriminator) that maximises L(G,D). Why is it not the other way around, i.e. argmax(argmin(L(G,...
0
votes
0
answers
17
views
Cost function looks like the real math which is responsible for actually working on out problem statement If I talk on a whole and on a surface level?
Looking at the cost function for say linear regression, apart from changing the weight or the parameters, the cost function does the real job, right? If it is correct, what does cost function do in ...
0
votes
0
answers
13
views
Search recall optimization - what appropriate loss function to use?
I am studying machine learning and wanted to work on a project of my own so that I have better chances after graduating college. I'm studying the application of ML to improve searches using a toy ...
- 167
0
votes
0
answers
205
views
Training loss is much higher than validation loss
I am trying to train a neural network with 2 hidden layers to perform a multi class classification of 3 different classes. There is a huge imbalance to the classes, with the distribution being around ...
1
vote
0
answers
19
views
Avoid overfitting to noise by a noise penalty approach instead of early stopping?
I came across this article on deep learning for computational MRI and found an interesting sentence "However, early stopping has to be performed to not overfit to the noisy measurements." ...
- 11
0
votes
0
answers
15
views
Loss function for classifcation rewarding closer guess?
The default loss function in multi class classification is cross_entropy, which treats all wrong guesses equally. If the distance between buckets are meaningful, for example, given the real bucket is ...
- 101
2
votes
1
answer
165
views
What is the benefit of the exponential function inside softmax?
I know that softmax is:
$$ softmax(x) = \frac{e^{x_i}}{\sum_j^n e^{x_j}}$$
This is an $\mathbb{R}^n \implies \mathbb{R}^n$ function, and the elements of the output add up to 1. I understand that the ...
- 123
3
votes
1
answer
100
views
Representation of Strictly Proper Scoring Rule for Multiclasss Classificaiton
I am working on a classification problem, using features $\mathbf{x}$ to predict a target variable $y \in \mathbb{N}_0$. By a strictly proper scoring rule I mean a loss function $\ell(y,\hat{y})$ for ...
- 103
1
vote
0
answers
88
views
Replication of XGBoost's binary:logistic loss
I am trying to replicate XGBoost's logistic loss function as a first step before implementing my own custom loss functions. Following from here and looking at the original code in git repository, I ...
- 111