Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.
69 votes
Accepted

Why do cost functions use the square error?

Your loss function would not work because it incentivizes setting $\theta_1$ to any finite value and $\theta_0$ to $-\infty$. Let's call $r(x,y)=\frac{1}{m}\sum_{i=1}^m {h_\theta\left(x^{(i)}\right)} ...
user avatar
  • 1,011
61 votes
Accepted

Sparse_categorical_crossentropy vs categorical_crossentropy (keras, accuracy)

Use sparse categorical crossentropy when your classes are mutually exclusive (e.g. when each sample belongs exactly to one class) and categorical crossentropy when one sample can have multiple classes ...
user avatar
45 votes
Accepted

What does from_logits=True do in SparseCategoricalcrossEntropy loss function?

The from_logits=True attribute inform the loss function that the output values generated by the model are not normalized, a.k.a. logits. In other words, the softmax ...
user avatar
  • 634
38 votes
Accepted

Intuitive explanation of Noise Contrastive Estimation (NCE) loss?

Taken from this post:https://stats.stackexchange.com/a/245452/154812 The issue There are some issues with learning the word vectors using an "standard" neural network. In this way, the word vectors ...
user avatar
34 votes

Sparse_categorical_crossentropy vs categorical_crossentropy (keras, accuracy)

The answer, in a nutshell If your targets are one-hot encoded, use categorical_crossentropy. Examples of one-hot encodings: ...
user avatar
  • 341
31 votes

Why do cost functions use the square error?

The 1/2 coefficient is merely for convenience; it makes the derivative, which is the function actually being optimized, look nicer. The 1/m is more fundamental; it suggests that we are interested in ...
user avatar
  • 10.4k
29 votes
Accepted

What is the advantage of using log softmax instead of softmax?

There are a number of advantages of using log softmax over softmax including practical reasons like improved numerical performance and gradient optimization. These advantages can be extremely ...
user avatar
  • 647
19 votes

L2 loss vs. mean squared loss

Function $L_2(x):=\left \|x \right \|_2$ is a norm, it is not a loss by itself. It is called a "loss" when it is used in a loss function to measure a distance between two vectors, $\left \| y_1 - y_2 \...
user avatar
  • 8,509
18 votes

Custom loss function with additional parameter in Keras

You can write a function that returns another function, as is done here on GitHub ...
user avatar
17 votes
Accepted

Parameterization regression of rotation angle

The second way, predicting $x=cos(\alpha)$ and $y=sin(\alpha)$ is totally okay. Yes, the norm of the predicted $(x, y)$ vector is not guaranteed to be near $1$. But it is not likely to blow up, ...
user avatar
  • 1,471
16 votes

Keras Sequential model returns loss 'nan'

To sum up the different solutions from both stackOverflow and github, which would depend of course on your particular situation: Add regularization to add l1 or l2 penalties to the weights. Otherwise,...
user avatar
  • 271
14 votes
Accepted

Why is there a $2$ at the denominator of the mean squared error function?

This is just for mathematical convenience. When you differentiate $C(w,b)$, you will get an extra $2$. To eliminate that, $2$ is kept beforehand in denominator. You can also watch this video on SVM ...
user avatar
  • 1,729
14 votes

What is the difference between SGD classifier and the Logisitc regression?

Welcome to SE:Data Science. SGD is a optimization method, while Logistic Regression (LR) is a machine learning algorithm/model. You can think of that a machine learning model defines a loss function, ...
user avatar
  • 2,119
13 votes
Accepted

Custom loss function with additional parameter in Keras

I think the best solution is: add the weights to the second column of y_true and then: ...
user avatar
  • 641
11 votes

Validation loss is lower than the training loss

It is certainly correct in the sense that it is a legitimate neural network. The dropout layer introduces noise that is not injected during the test period. The goal is to combat overfitting so that ...
user avatar
11 votes
Accepted

Why does putting a 1/2 in front of the squared error make the math easier?

A major reason for using MSE is to optimize the parameters of a regression model. From calculus, you know how to find the minimum of a function by taking the derivative. That puts a "2" out ...
user avatar
  • 2,745
10 votes

What's the difference between Error, Risk and Loss?

Error In this context, error is the difference between the actual / true value ($\theta$) and the predicted / estimated value ($\hat\theta$) $$Error = \theta - \hat\theta$$ Loss Loss and Risk are ...
user avatar
  • 281
10 votes
Accepted

Validation loss

Validation loss is the same metric as training loss, but it is not used to update the weights. It is calculated in the same way - by running the network forward over inputs $\mathbf{x}_i$ and ...
user avatar
  • 27.2k
9 votes
Accepted

Interpreting the Root Mean Squared Error (RMSE)!

How can I interpret RMSE? RMSE is exactly what's defined. $24.5 is the square root of the average of squared differences between your prediction and your actual observation. Taking squared ...
user avatar
  • 3,358
8 votes

L2 loss vs. mean squared loss

To be precise, L2 norm of the error vector is a root mean-squared error, up to a constant factor. Hence the squared L2-norm notation $\|e\|^2_2$, commonly found in loss functions. However, $L_p$-...
user avatar
  • 355
8 votes

What Base Should Be Used For Negative Log Likelihood?

The change in base is equivalent to multiplying the function by a constant. It does not affect the computation. $ log_b(x) = \dfrac{1}{log_e(b)}.log_e(x) $
user avatar
  • 535
7 votes

Why do cost functions use the square error?

The error measure in the loss function is a 'statistical distance'; in contrast to the popular and preliminary understanding of distance between two vectors in Euclidean space. With 'statistical ...
user avatar
7 votes
Accepted

Does keras categorical_cross_entropy loss take incorrect classification into account

Does this mean the errors for $y_i=0$ do not contribute to the loss? That is correct. However, the respective weights that connect to wrong neurons will still have gradients due to the error, and ...
user avatar
  • 27.2k
6 votes
Accepted

What are the differences between logistic and linear regression?

As you have mentioned, the output of linear regression is a real value while logistic regression's represents classes(classification). Their main difference is this. The loss function of linear ...
user avatar
  • 13.3k
6 votes

What is the advantage of using log softmax instead of softmax?

Log softmax is $$\log(\exp(x)/\sum(\exp(x))) =x - \log(\sum(\exp(x))).$$ Now $\log(\sum(\exp(x))) \approx \max(x)$, since the sum is dominated by the largest entry. We see that log softmax is nearly ...
user avatar
6 votes

loss/val_loss are decreasing but accuracies are the same in LSTM!

I think this is because your targets y are continuous instead of binary. Therefore, either ignore the accuracy report, or ...
user avatar
  • 8,509
6 votes

What makes binary cross entropy a better choice for binary classification than other loss functions?

As mentioned in the blog, cross entropy is used because it is equivalent to fitting the model using maximum likelihood estimation. This on the other hand can be interpreted as minimizing the ...
user avatar
  • 5,959
6 votes
Accepted

Does small batch size improve the model?

In general smaller or larger batch size doesn't guarantee better convergence. Batch size is more or less treated as a hyperparameter to tune keeping in the memory constraints you have. There is a ...
user avatar
  • 1,216
5 votes
Accepted

Keras categorical_crossentropy loss (and accuracy)

I am using keras with tensorflow backend. I checked and the categorical_crossentropy loss in keras is defined as you have defined. This is the part of code (not the whole function definition)- ...
user avatar
  • 1,729
5 votes

Why do cost functions use the square error?

In addition to the key points made by others, using squared error puts a greater emphasis on larger error (what happens to 1/2 when you square it vs 3/2?). Having an algorithm that moves the ...
user avatar
  • 51

Only top scored, non community-wiki answers of a minimum length are eligible