Questions tagged [loss-function]
The loss-function tag has no usage guidance.
1
vote
2answers
65 views
What are the differences between logistic and linear regression?
I know that linear regression does "regression" and logistic regression does "classification". When we implement these two methods, the only difference I could notice is the loss function: linear ...
2
votes
1answer
11 views
Loss function for stoichiometry balancing
I'm looking for a differentiable loss function to estimate the cost of unbalance in protein synthesis.
For example, there are three subunits which have concentrations of 1, 2, 1 respectively.
The ...
1
vote
1answer
11 views
Constant validation loss & accuracy, training accuracy fluctuates
I am training a Squeeze-net model for binary classification of images. I have 79968 images for training (50:50 for and against) and 8892 images in the validation set. After 35000 iterations my ...
1
vote
1answer
25 views
What is the difference between SGD classifier and the Logisitc regression?
To my understanding, the SGD classifier, and Logistic regression seems similar.
An SGD classifier with loss = 'log' implements Logistic regression and loss = 'hinge' implements Linear SVM. I also ...
1
vote
1answer
64 views
Purpose of backpropagation in neural networks
I've just finished conceptually studying linear and logistic regression functions and their optimization as preparation for neural networks.
For example, say we are performing binary classification ...
0
votes
0answers
21 views
Train the model to increase accuracy rather than to minimise loss
I am currently in a situation of seq2seq training where the cross entropy loss is very low (near zero) but the accuracy is also very low. This made me wondering if there were any loss functions that ...
1
vote
1answer
19 views
Validation loss
I am having trouble wrapping my brain around validation loss. It's my understanding that loss is calculated at the end of the feed forward in a NeuralNet and is used in back propagation to update the ...
1
vote
0answers
33 views
Why is my loss function for DQN converging too quickly?
I'm still relatively new to deep learning and am experiencing an issue that I can't seem to find a solution/explanation for. I've developed a DQN model in tensorflow, as described by DeepMind, and am ...
2
votes
0answers
16 views
3
votes
1answer
48 views
Interpreting the Root Mean Squared Error (RMSE)!
I real all about pros and cons of RMSE vs. other absolute errors namely mean absolute error (MAE). See the the following references:
MAE and RMSE — Which Metric is Better?
What's the bottom line? How ...
2
votes
1answer
25 views
Why are optimization algorithms slower at critical points?
I just found the animation below from Alec Radford's presentation:
As visible, all algorithms are considerably slowed down at saddle point (where derivative is 0) and quicken up once they get out of ...
2
votes
0answers
20 views
Validation loss keeps fluctuating about training loss
I am training a Keras model for multi-target regression by using a custom loss function with the goal of getting predictions accurate to below 0.01 with respect to ...
0
votes
1answer
20 views
Differentiating roadmap of a loss function
Let's say I'm performing Stochastic Gradient Descent (SGD) on binary cross entropy error while optimizing weight $w_{2}$.
Binary cross entropy error:
$$L(y|p(x_{i}))=-y_{i}*ln(p(x_{i}))-(1-y_{i})*...
1
vote
0answers
20 views
Deep RL: Proximal policy optimization gradient calculation
Case:
Continuous action domain with 4 outputs (control problem)
Policy and Value function approximation with fully connected neural networks
I understand that the loss function for PPO for the ...
-1
votes
1answer
27 views
What's the difference between Error, Risk and Loss?
When we talk about 'Minimizing Loss', we often talk about loss functions such as Mean Squared Error (MSE); the term 'Empirical Risk Minimization' is often used interchangeably. So what's the ...
0
votes
0answers
12 views
Keras update Layers which are only Part of the Loss Function
I'm implementing a variational inference model. There are two main structures a inference network and a generative model. The first part is only used for training and only appears in the loss function....
1
vote
1answer
150 views
focal loss function help
I am working on a relation extraction and classification problem. The data is in the form of text files. The data is imbalanced. I want to use focal loss function to address class imbalance problem in ...
0
votes
2answers
305 views
Custom loss function which is included gradient in Keras
I want to make a custom loss function.
Concretely, I use a 2D Convolutional neural network in Keras.
So far, I've made various custom loss function by adding to losses.py.
However, in this case, I ...
1
vote
2answers
112 views
How the combination of cross entropy loss and gradient descent penalizes and rewards
For a simple problem of classification (C classes) using the softmax classifier, most people use the cross-entropy loss function to quantify the objective. The cross-entropy loss is:
$$L = -\sum_{i=1}...
0
votes
1answer
147 views
Loss function for an RNN used for binary classification
I'm using an RNN consisting of GRU cells to compare two bounding box trajectories and determine whether they belong to the same agent or not. In other words, I am only interested in a single final ...
0
votes
1answer
17 views
Loss function range normalization
This is from a referee report in a conference to which I submitted my paper - I don't quite get it and I'm not sure what I need to do about it.
I use Euclidean loss and Softmax cross-entropy (...
0
votes
0answers
102 views
Tensorflow - Clarification on how to predict more than 1 step ahead using LSTM
I'm looking for some clarification on how to perform more than one prediction ahead using long-short term memory in Tensorflow for univariate time series. The reason is that I'm having problems when I ...
1
vote
0answers
13 views
neural network function approximation with constraints
I would like to approximate a function $f(\cdot)$ by means of a neural network given a finite set of observations $f(x_i)$ where $x_i\in\mathbb{R}^n$ and $i=1\dots,N$. However, I have some prior ...
0
votes
0answers
46 views
Can a Warm-up loss period easily be implemented in Keras?
I am creating a RNN in Keras. It was suggested that I utilize a warm-up period before loss is calculated to increase accuracy down the line.
I saw some people achieved this by creating a loss ...
0
votes
0answers
10 views
Decision Tree problem with dynamic loss function
I have a marketing problem, we have service lines, and margin on each line.
For lines that disconnect and lines that do not, we want to identify features that help us maximize the difference between ...
3
votes
1answer
35 views
Can pinball loss be used to construct a prediction interval?
I'm modeling some time series data ($\{y_t\}_t$) and would like to construct a model that is able to return not just a single-value prediction $\hat{y_t}$, but an interval $C_t=(\hat{y}_{t, lower}, \...
3
votes
0answers
198 views
Precision recall loss function
I've been using precision and recall as my metrics, as per keras-team/keras/pull/9393/files
Sensitivity & specificity is what I want to optimise for. Every epoch I output it:
...
2
votes
1answer
116 views
Understanding LSTM behaviour: Validation loss smaller than training loss throughout training for regression problem
I'm building a lstm model for regression on timeseries. To verify my implementation of the model and understand keras, I'm using a toyproblem to make sure I understand what's going on. Problem is I do ...
1
vote
1answer
250 views
How to create a custom loss function from sklearn metrics in Keras? [closed]
I'd like to use the mutual information metric from sklearn as a loss function for a neural network in Keras, but I'm not sure how to do it. I'd like to try this because relationships in my dataset are ...
4
votes
2answers
107 views
What algorithms will stuck in the local minimum?
Algorithms like neural network are easily getting stuck in local minimum because the shape of the loss function (so there are parameters like momentum are designed to solve this type of problem). ...
1
vote
2answers
19 views
Weighting influence of two neural networks on classification
I'm training a model that has two neural networks. One of them is a resnet18 CNN which has as it's input images. The other one is a small one hidden layer network that has as it's input four other ...
2
votes
0answers
30 views
Cross error loss function cause division by zero error
How to calculate cross entropy when actual output is 0? Would not it give indf brcause of log(0) and the cross entropy for binary classification is given by:
-(ylog(actual_output)+(1-y)*(1-...
3
votes
2answers
75 views
Why validation loss worsens while precision/recall continue to improve?
I'm training a neural network on 'easy' dataset with ~15k examples.
Network overfits pretty fast.
The thing I cannot understand that after 5th epoch validation loss is starting to worsen, while ...
1
vote
1answer
37 views
Loss for CNN decreases and settles but training accuracy does not improve
I am training a CNN with 2 conv layers 2 Relu and max pooling and 2 FC layers the last of which has only 2 units since it's a binary classification problem. The images are spatio-temporal continuous, ...
2
votes
0answers
78 views
How to use a cross entropy loss function for each letter/digit in a captcha?
I'm trying to develop a captcha solver using a simple fully-connected neural network in TensorFlow. All captchas have 5 digits/letters, each character can be a number 0-9 or a letter A-Z. They all ...
1
vote
1answer
46 views
Difference between mathematical and Tensorflow implementation of Softmax Crossentropy with logit
Softmax cross entropy with logits can be defined as:
$a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$
$l={\sum_{\forall i}}y_ilog(a_i)$
Where $l$ is the actual loss.
But when you look deep inside ...
2
votes
0answers
170 views
Keras/TensorFlow in R - Additional Vector to Custom Loss Function
I would like to pass a vector that is outside of the training data, but the same length as the training data, to a custom loss function.
The vector represents a post-prediction funnel (one or zero) ...
2
votes
2answers
40 views
Classifier that optimizes performance on only a subset of the data?
I'm working on machine learning problem where I'm only interested in getting high accuracy within a narrow band of my predicted likelihoods. Specifically, I want an algorithm that will score very ...
7
votes
2answers
110 views
Validation showing huge fluctuations. What could be the cause?
I'm training a CNN for a 3-class image classification problem. My training loss decreased smoothly, which is the expected behaviour. However, my validation loss shows a lot of fluctuation.
Is this ...
3
votes
1answer
104 views
Train loss vs validation loss
I have a few basic questions about tracking losses during training.
If I am using mini-batch training, should I validate after each
batch update or after I have seen the entire dataset?
What should ...
2
votes
1answer
21 views
Class distribution discrepancy training/validation. Loss now uninterpretable?
I have a 3-class image classification problem. The classes are highly unbalanced (with about 98% of the images beloning to one class). To counteract this unbalanced data, I weight the losses by using ...
0
votes
0answers
135 views
Problem with Loss function in CNN for Object Detection
I am trying to train a CNN in TensorFlow to perform text localization in images by using regression to output a bounding box around the text. I have created a custom dataset by pasting images from the ...
0
votes
0answers
16 views
Equation of SphereFace
I have a question about the sphere loss. For $k∈[0,m−1], m≥1$ we define
$$ψ(θ_{y_i,i})=(−1)^k\cos(mθ_{y_i,i})−2k, \textrm{where} \, θ_{y_i,i} \in [kπ/m,(k+1)π/m]$$
Why we can do this, and what kind ...
0
votes
0answers
45 views
NN: choose a loss function for training
I am training a neural network for action units detection. And I have a problem defining a good loss function.
The neural network must detect 39 action units if they occur in a picture. So the output ...
2
votes
0answers
162 views
Cost/loss functions for multi-tasking regression neural networks
The mean square loss function is the standard for regression neural networks. However, if I have a neural network learning two tasks (two outputs) at once, is it more advisable to train on the sum of ...
0
votes
1answer
386 views
What loss function should I use for auto encoders for text
I'm trying to implement an autoencoder for text. But I don't know which loss function I should use ? I tried using the mse but I get a huge loss 1063442. I'm trying to build a very simple autoencoder ...
0
votes
2answers
108 views
How to use the finite difference to compute gradient for very complex loss function
I need to put the prediction into a complex function to calculate loss.
This means that i can't build the loss function by tensorflow's operator and can't get gradient from Automatic Differentiation.
...
1
vote
2answers
351 views
Mean Squared Error in the Deep Learning Book
In the famous Deep Learning Book, in chapter 1, equation 6, the Quadratic Cost (or Mean Squared Error) in a neural network is defined as
$ C(w, b) = \frac{1}{2n}\sum_{x}||y(x)-a||^2 $
where $w$ is ...
0
votes
0answers
163 views
Loss function for an anomaly detection system with LSTM
I'm working with a time series dataset of categorical signal and numerical data (200000,50).
I use all signals to train an LSTM model and then I use the prediction error to find anomalies.
I've build ...
1
vote
1answer
48 views
How to use correct weights in linear regression model
I'm trying to understand can we implement a simple linear regression model.
Let's say we are predicting price currencies. We want to know whether the currency will raise or not.
As i understand, we ...
