19

Let's first see what we need to do when we want to train a model. First, we want to decide a model architecture, this is the number of hidden layers and activation functions, etc. (compile) Secondly, we will want to train our model to get all the paramters to the correct value to map our inputs to our outputs. (fit) Lastly, we will want to use this model ...


16

It is simple. It is because when you take the derivative of the cost function, that is used in updating the parameters during gradient descent, that $2$ in the power get cancelled with the $\frac{1}{2}$ multiplier, thus the derivation is cleaner. These techniques are or somewhat similar are widely used in math in order "To make the derivations mathematically ...


13

In this case, the two math formulae show you the correct type of multiplication: $y_i$ and $\text{log}(a_i)$ in the cost function are scalar values. Composing the scalar values into a given sum over each example does not change this, and you never combine one example's values with another in this sum. So each element of $y$ only interacts with its matching ...


10

Ok, so here is how it works. Say you want to classify animals, and you have cats, dogs and birds. This means that your model will output 3 units in the form of a vector (call it a list if you prefer). Each element of the list represents and animal, so for example Position 0 represents how likely the input is to be a cat Position 1 represents how likely ...


9

First, remember that the derivative of a function gives the direction in which the function increases, and its negative, the direction in which the function decreases. Training a model is just minimising the loss function, and to minimise you want to move in the negative direction of the derivative. Back-propagation is the process of calculating the ...


8

Another approach was suggested in this paper for face age estimation: https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Niu_Ordinal_Regression_With_CVPR_2016_paper.pdf The authors use a number of binary classifiers predicting whether a data point is larger than a threshold, and do this for multiple thresholds. I.e. in your case the network ...


7

Whenever you have a convex cost function you are allowed to initialize your weights to zeros. The cost function of logistic regression and linear regression have convex cost function if you use MSE for, also RSS, linear regression and cross-entropy for logistic regression. The main idea is that for convex cost function you'll have just a single optimal point ...


7

The loss functions are only simple convex functions with respect to the weight parameters (and specific data) when there is a single layer. More exactly, they can proven to be always convex with respect to the weights in the simple models (linear or logistic regression), but not with respect to weights of deeper networks. You can prove that there must be ...


5

The author has taken activation function as sigmoid in this case. The derivative of this function can be re-written like $$\sigma(z) = \frac{1}{1+e^{-x}}$$ whose derivative $$\sigma'(z) = \frac{e^{-x}}{(1+e^{-x})^2}$$ which can rewritten as $$\frac{1+e^{-x}-1}{(1+e^{-x})^2} \rightarrow \sigma(z) * (1 - \sigma(z))$$. This is subtly mentioned in eq.3 of ...


5

The cost function of the Logistic Regression derived via Maximum Likelihood Estimation: If y = 1 (positive): i) cost = 0 if prediction is correct (i.e. h=1), ii) cost $\rightarrow \infty $ if $h_{\theta}(x)\rightarrow 0$. If y = 0 (negative): i) cost = 0 if prediction is correct (i.e. h=0), ii) cost $\rightarrow \infty$ if $(1-h_{\theta}(x))\rightarrow ...


5

It makes the math easier to handle. Adding a half or not doesn't actually matter since minimizing is unaffected by constants.


5

There can be other reasons related to the model but the most simple explanation is that the data contains contradicting patterns: if the same features correspond to different target values, there is no way for any model to achieve perfect performance. Let me illustrate with a toy example: x y 0 4 0 4 1 3 1 4 1 3 1 3 2 8 2 7 2 7 2 6 ...


4

The weights in a model do not need to converge to stop training. One possible explanation is that the model error surface has a big, wide valley. If that is the case, the loss function would be low throughout the valley but there would be many weight combinations that would all yield similar performance on the training dataset. Once a model has reached an ...


4

Not clear if you are saying the cost of a FN or a FP is higher, you only mention FN in your statement. Think that you mean a FN is more costly and that a positive means a 1. In general, if an incorrect prediction for the minority case is more costly (FN), you should sample that minority case higher or the majority case lower so the ratio is closer to 1:1. ...


4

tl;dr since you are referring to accuracy, I'm guessing you have a classification task. There is no way to evaluate a model's actual performance, by its loss. The goal of the loss function is to train the model and not to show how well this model classifies. What you should do to see the best model is to evaluate it by a classification metric (e.g. accuracy)....


3

Not trying to oversimplify the answer, but simply get a calculator to compute these manually and you can see this in action: If t is close to 1, lets just say that is 0.9999 for the example, then: $$ -log(t) = -log(0.9999) = 0.000100005 $$ conversely, If t is close to 0, lets just say that is 0.0001 for the example, then: $$ -log(t) = -log(0.0001) = 9....


3

Saying that the well-known loss functions, like MSE or Categorical Cross Entropy, has a global minimum value equal to zero is flawed . The idea behind loss function is to measure how near the model predictions are to the actuals(in case of a regression). Now ideally , you would want your model to predict exactly equal to the actuals . In that case only , we ...


3

This is really just like a convention that appears in some places because we normallt want to take the derivative of the cost function (i.e. compute gradients), which means the power of 2 would be taken to the front. If we put the $\frac{1}{2}$ at the front to begin with, it just looks nicer once we have finished. I have seen this written somewhere before ...


3

In a scenario where consequences of prediction errors are not equivalent, you are usually still interested in training a model to predict accurately from the data set, and would not change the objective function in supervised learning. Typically when consequences of FP and FN differ, you would: Use the confidence of the prediction given by the model. In ...


3

Gradient Descent is an universal method, you can us it with basically every loss function you can find in known ML algorithms. In your case, you have only to derive the logarithmic cost function. You can find a detailed calculation at https://math.stackexchange.com/questions/477207/derivative-of-cost-function-for-logistic-regression


3

What you need to use is mini-batch gradient descent or stochastic gradient descent. You will need to shuffle your samples and make draws from it of the batch size you are aiming for, also you will have to make sure all of the samples in your data are included which will constitute of one epoch. Train for a few epochs depending on the data. Here is a good ...


3

There are two types of classification costs: per class, and per instance. In keras, for instance cost, we assign a cost to each training sample by feeding sample_weight to .fit. For example, if we have four training samples in rows 1, 2, 3, and 4, with misclassification costs 2.5, 1.5, 1.0, and 1.0, we feed sample_weight=[2.5, 1.5, 1.0, 1.0]. For class ...


3

Supporting the second answer by Donald, I would also try to look at the predicted probabilities (via the predict_proba attribute in scikit learn classifiers) and customize your predictions by selecting the threshold (which you can check also in the corresponding ROC curve) which gives you the most robust (i.e. highest) predicted probabilities distributions, ...


2

You are talking about having different missclassification costs. Classification consists of outputing a continuous score which is then thresholded to decide the discrete classification. The threshold (ex. probability>0.5 leads to C) is chosen to satisfy your apetite on sensitivty/specificity. The training algorithm only cares about your continuous score. ...


2

The following piece of code does essentially what TF's softmax_cross_entropy_with_logits functions does (crossentropy on softmaxed y_ and y): import scipy as sp import numpy as np def softmax(x): e_x = np.exp(x - np.max(x)) return e_x / e_x.sum(axis=0) def crossentropy(true, pred): epsilon = 1e-15 pred = sp.maximum(epsilon, pred) ...


2

Are you asking, what's the difference between a dot product of two vectors, and summing their elementwise product? They are the same. np.sum(X * Y) is np.dot(X, Y). The dot version would be more efficient and easy to understand, generally. But in the cost function, $Y$ is a matrix, not a vector. np.dot actually computes a matrix product, and the sum of ...


2

If you feed the output of the LSTM directly into a softmax, you probably won't get good results. If you use a softmax layer after a tanh layer, bad stuff happens. As you say, the confidence will never get near 100%. For instance, if there are two classes, you can never get above about 88% confidence. If there are $k$ classes, you can never get ...


2

If want you mean by 100% certainty is: 1, 0, 0, 0, as by your example, you cannot obtain this type of output using soft-max. For this to happen you will need that the softmax equals 0 for those i that are not correct, and this simply cannot happen because: $$ e^{h_i} \neq 0 $$ $$ \forall h_i \in \mathbb{R} $$ See this link for better explanation. ...


2

As stated in the last edit of my question, the issue indeed was to do with the softmax function. As clarified here We shouldn't apply softmax directly to the result of the last LSTM. Notice, LSTM will produce a vector of values, each of which is bounded between -1 and 1 (due to the tanh squashing function that's applied to the Cell). Instead, I've created ...


2

Zeroing weights disables them. Yes, there are various applications of zero tensors (such as convex cost functions as you mention). Let's take the case of Neural Nets (NNs) and see if the math gives us more intuition: $$ tensor\div0 = undefined\\ tensor * 0 = 0\\ tensor \cdot 0 = 0\\ $$ Example Graph #1: How would one disable a single synapse connected to ...


Only top voted, non community-wiki answers of a minimum length are eligible