72
votes
Accepted
Why do cost functions use the square error?
Your loss function would not work because it incentivizes setting $\theta_1$ to any finite value and $\theta_0$ to $-\infty$.
Let's call $r(x,y)=\frac{1}{m}\sum_{i=1}^m {h_\theta\left(x^{(i)}\right)} ...
- 1,041
67
votes
Accepted
Sparse_categorical_crossentropy vs categorical_crossentropy (keras, accuracy)
Use sparse categorical crossentropy when your classes are mutually exclusive (e.g. when each sample belongs exactly to one class) and categorical crossentropy when one sample can have multiple classes ...
- 786
65
votes
Accepted
What does from_logits=True do in SparseCategoricalcrossEntropy loss function?
The from_logits=True attribute inform the loss function that the output values generated by the model are not normalized, a.k.a. logits. In other words, the softmax ...
- 854
42
votes
Accepted
Intuitive explanation of Noise Contrastive Estimation (NCE) loss?
Taken from this post:https://stats.stackexchange.com/a/245452/154812
The issue
There are some issues with learning the word vectors using an "standard" neural network. In this way, the word vectors ...
- 544
38
votes
Sparse_categorical_crossentropy vs categorical_crossentropy (keras, accuracy)
The answer, in a nutshell
If your targets are one-hot encoded, use categorical_crossentropy.
Examples of one-hot encodings:
...
- 381
36
votes
Accepted
What is the advantage of using log softmax instead of softmax?
There are a number of advantages of using log softmax over softmax including practical reasons like improved numerical performance and gradient optimization. These advantages can be extremely ...
- 717
31
votes
Why do cost functions use the square error?
The 1/2 coefficient is merely for convenience; it makes the derivative, which is the function actually being optimized, look nicer. The 1/m is more fundamental; it suggests that we are interested in ...
- 10.5k
28
votes
L2 loss vs. mean squared loss
Function $L_2(x):=\left \|x \right \|_2$ is a norm, it is not a loss by itself. It is called a "loss" when it is used in a loss function to measure a distance between two vectors, $\left \| y_1 - y_2 \...
- 9,247
21
votes
Keras Sequential model returns loss 'nan'
To sum up the different solutions from both stackOverflow and github, which would depend of course on your particular situation:
Check validity of inputs (no NaNs or sometimes 0s). i.e df.isnull()....
- 351
19
votes
Custom loss function with additional parameter in Keras
You can write a function that returns another function, as is done here on GitHub
...
- 9,348
18
votes
Accepted
Parameterization regression of rotation angle
The second way, predicting $x=cos(\alpha)$ and $y=sin(\alpha)$ is totally okay.
Yes, the norm of the predicted $(x, y)$ vector is not guaranteed to be near $1$. But it is not likely to blow up, ...
- 1,541
16
votes
Accepted
Why is there a $2$ at the denominator of the mean squared error function?
This is just for mathematical convenience. When you differentiate $C(w,b)$, you will get an extra $2$. To eliminate that, $2$ is kept beforehand in denominator.
You can also watch this video on SVM ...
- 1,821
15
votes
Accepted
What is the difference between SGD classifier and the Logisitc regression?
Welcome to SE:Data Science.
SGD is a optimization method, while Logistic Regression (LR) is a machine learning algorithm/model. You can think of that a machine learning model defines a loss function, ...
- 2,214
15
votes
What is the advantage of using log softmax instead of softmax?
Log softmax is
$$\log(\exp(x)/\sum(\exp(x)))
=x - \log(\sum(\exp(x))).$$
Now $\log(\sum(\exp(x))) \approx \max(x)$, since the sum is dominated by the largest entry.
We see that log softmax is nearly ...
- 261
14
votes
Accepted
Custom loss function with additional parameter in Keras
I think the best solution is:
add the weights to the second column of y_true
and then:
...
- 661
12
votes
What's the difference between Error, Risk and Loss?
Error
In this context, error is the difference between the actual / true value ($\theta$) and the predicted / estimated value ($\hat\theta$)
$$Error = \theta - \hat\theta$$
Loss
Loss and Risk are ...
- 320
11
votes
Accepted
Validation loss
Validation loss is the same metric as training loss, but it is not used to update the weights. It is calculated in the same way - by running the network forward over inputs $\mathbf{x}_i$ and ...
- 28.6k
11
votes
Validation loss is lower than the training loss
It is certainly correct in the sense that it is a legitimate neural network. The dropout layer introduces noise that is not injected during the test period. The goal is to combat overfitting so that ...
- 9,348
11
votes
Accepted
Why does putting a 1/2 in front of the squared error make the math easier?
A major reason for using MSE is to optimize the parameters of a regression model. From calculus, you know how to find the minimum of a function by taking the derivative. That puts a "2" out ...
- 3,909
11
votes
Why using a partial derivative for the loss function?
It’s a minimization problem. The typical calculus approach is to find where the derivative is zero and then argue for that to be a global minimum rather than a maximum, saddle point, or local minimum.
...
- 3,909
10
votes
L2 loss vs. mean squared loss
To be precise, L2 norm of the error vector is a root mean-squared error,
up to a constant factor. Hence the squared L2-norm notation $\|e\|^2_2$, commonly found in loss functions.
However, $L_p$-...
- 385
10
votes
L2 loss vs. mean squared loss
They are different:
L2 = $\sqrt{\sum_{i=1}^{N}(y_i-y_{i}^{pred})^2}$
MSE = $\frac{\sum_{i=1}^{N}(y_i-y_{i}^{pred})^2}{N}$
There are sum and ...
- 201
10
votes
Accepted
Interpreting the Root Mean Squared Error (RMSE)!
How can I interpret RMSE?
RMSE is exactly what's defined. $24.5 is the square root of the average of squared differences between your prediction and your actual observation. Taking squared ...
- 3,520
8
votes
Why do cost functions use the square error?
The error measure in the loss function is a 'statistical distance'; in contrast to the popular and preliminary understanding of distance between two vectors in Euclidean space. With 'statistical ...
- 1,313
8
votes
What Base Should Be Used For Negative Log Likelihood?
The change in base is equivalent to multiplying the function by a constant. It does not affect the computation.
$
log_b(x) = \dfrac{1}{log_e(b)}.log_e(x)
$
- 535
8
votes
Accepted
Does small batch size improve the model?
In general smaller or larger batch size doesn't guarantee better convergence. Batch size is more or less treated as a hyperparameter to tune keeping in the memory constraints you have.
There is a ...
- 1,346
8
votes
Is Pearson correlation a good loss function?
UPDATE (I WAS WRONG)
Maximizing correlation misses a lot and makes for a terrible function to optimize.
For instance, correlation will not detect if you consistently predict too high or too low.
For ...
- 3,909
7
votes
Accepted
Does keras categorical_cross_entropy loss take incorrect classification into account
Does this mean the errors for $y_i=0$ do not contribute to the loss?
That is correct.
However, the respective weights that connect to wrong neurons will still have gradients due to the error, and ...
- 28.6k
6
votes
Accepted
What are the differences between logistic and linear regression?
As you have mentioned, the output of linear regression is a real value while logistic regression's represents classes(classification). Their main difference is this.
The loss function of linear ...
- 13.9k
6
votes
Is empirical risk the same thing as loss function?
I found the accepted answer pretty hard to understand. Here is a simplified version that cleared it up for me:
Loss Function: a loss or risk function, $\mathcal{L}(\hat{y}^i, y^i)$, quantifies how ...
- 201
Only top scored, non community-wiki answers of a minimum length are eligible