Questions tagged [cost-function]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
4
votes
2answers
30 views

Question of using gradient descent instead of calculus. I checked previous questions there are still points to clarify

First of all I checked http://stats.stackexchange.com/questions/23128/solving-for-regression-parameters-in-closed-form-vs-gradient-descent, http://stackoverflow.com/questions/26804656/why-do-we-use-...
1
vote
1answer
19 views

Reason behind the sum of rate factors for calculating cost function derivative

Suppose we have a network of neurons like below: We make a little change in weight w[l][j][k] on our network, and it can make change on our cost function from ...
0
votes
0answers
10 views

Minimum cost is not zero when calculating cross-entropy on soft labels

I am training a neural network using batches of soft labels, e.g. y = [[0.00, 0.25, 0.25, 0.50], ... [0.75, 0.00, 0.20, 0.05]] However, as ...
0
votes
0answers
12 views

Understanding about RNN loss

Problem I am now taking Andrew Ng's deep learning course on Coursera. Everything is great but when it comes to RNN, I sometimes feel confused. Here is a question about RNN (or more specifically, the ...
3
votes
2answers
52 views

Two different cost functions for neural networks, how they can give the same result?

One is: $$J=-\frac{1}{m}\sum_{i=1}^{m}\sum_{k=1}^{K}\Big[y_{k}^{i}\log\big((h_{\theta}(x^{i}))_k\big)+(1-y_{k}^{i})\log\big(1-(h_{\theta}(x^{i}))_k\big)\Big]$$ The other one is: $$J=-\frac{1}{m}\sum_{...
1
vote
2answers
29 views

Cost function - ideas

I build xgboost model for regression problem. By the default xgboost optimize $(y - y_{pred})^2$, so the RMSE will be the best eval metric to measure performance. But my task is to build the best ...
0
votes
3answers
29 views

How does cost function change by choice of activation function (ReLU, Sigmoid, Softmax)?

I am new to ML and as I take courses for the area DL, I am wondering, by our choice of activation function for the last layer, whether we take sigmoid, relu or softmax, would the formula for ...
0
votes
0answers
14 views

What's the correct way of implementation of cost function and gradient function in logistical regression after regularisation?

This is the cost function of logistic regression: which i could implement correctly, with the code : ...
5
votes
2answers
357 views

Why do we have to divide by 2 in the ML squared error cost function? [duplicate]

I'm not sure why you need to multiply by $\frac1{2m}$ in the beginning. I understand that you would have to divide the whole sum by $\frac1{m}$, but why do we have to multiply $m$ by two? Is it ...
1
vote
1answer
18 views

Decision Tree Optimize Deviation From Objective

I have the following problem: I have three classes/modes, let's call them car, bike, and walking. For any given test data instance with some environmental variables such as distance, road quality etc, ...
1
vote
1answer
24 views

How is the linear regression cost function evolved?

A couple of weeks ago I joined the Standford University machine learning course on Coursera. In that course, they directly gave the cost function formula without telling how this formula was evolved. ...
2
votes
1answer
359 views

Cost sensitive classification with individual cost

I'm currently sitting on a problem, where i'm uncertain if there is not a much simpler solution. I'm trying to train a DNN with a dataset for a classification task that should be cost sensitive. ...
1
vote
1answer
37 views

Logitic Regression cost function - what if ln(0)?

I am building logistic regression from scrap. The simplified cost function I am using is (from machine learning course on coursera): in specific case during learning, one observation in training ...
2
votes
1answer
56 views

Understanding minimizing cost correctly

I cannot wrap my head around this simple concept. Suppose we have a linear regression, and there is a single parameter theta to be optimized (for simplicity purposes): $h(x) = \theta \cdot x$ The ...
5
votes
1answer
1k views

What do “compile” , “fit” and “predict” do in Keras sequential models?

I am a little confused between these two parts of Keras sequential models functions. May someone explains what is exactly the job of each one? I mean ...
0
votes
0answers
14 views

cost/loss function being a multi-well function in neural networks

In a 0 or 1 binary classification problem using neural networks, given the activation function is taken as sigmoid, if one takes the cost/loss function as sum of square of differences, the loss ...
3
votes
4answers
123 views

Weights not converging while cost function has converged in neural networks

My cost/loss function drops drastically and approaches 0, which looks a sign of convergence. But the weights are still changing in a visible way, a lot faster than the cost function. Should I ensure ...
3
votes
1answer
327 views

How does Gradient Descent and Backpropagation work together?

Please forgive me as I am new to this. I have attached a diagram trying to model my understanding of neural network and Back-propagation? From videos on coursera and resources online I formed the ...
5
votes
2answers
140 views

What is an intuitive explanation for the log loss cost function?

I would really appreciate if someone could explain the log loss cost function And the use of it in measuring a classification model performance. I have read a few articles but most of them ...
3
votes
2answers
178 views

Logistic regression cost function

In Aurelien Geron's book I found this line ...
0
votes
1answer
643 views

Should the minimum value of a cost (loss) function be equal to zero?

We know optimization techniques search in the space of all the possible parameters for a parameter set that minimizes the cost function of the model. The most well-known loss functions, like MSE or ...
0
votes
1answer
65 views

How to Define a Cost Fucntion?

I want to define a cost function in python to identify optimum value in days when i should end a marketing campaign to save spend on campaigns not generating traffic good traffic. Problem is I dont ...
2
votes
1answer
249 views

boosting an xgboost classifier with another xgboost classifier using different sets of features

What I would like to do, is train a first model $f_{1}(\underline{x})$, where $\underline{x}$ is a set of features, fix what model 1 has learned, and then train a second model $f_{2}(\underline{y})$ ...
2
votes
0answers
116 views

How to resolve the instability of average reward per episode in training of DQN (Deep Q-Network)?

what is shown when average reward per episode in training is unstable? If there is big difference between average reward per episode and final reward by test section, what we can say? For ...
0
votes
1answer
101 views

Policy Gradient Methods - ScoreFunction & Log(policy)

In Policy Gradient Methods, Lecture 7 (34:15), David describes a Score Function as being the Gradient of the Log of the policy Question: If we have a Neural ...
0
votes
1answer
355 views

Should the cost function be zero using TensorFlow's sigmoid_cross_entropy_with_logits?

I'm building a CNN to make a binary classification (1 or zero). For this I'm using the cost function sigmoid_cross_entropy_with_logits. But for some reason the ...
3
votes
1answer
2k views

What is the Time Complexity of Linear Regression?

I am working with linear regression and I would like to know the Time complexity in big-O notation. The cost function of linear regression without an optimisation algorithm (such as Gradient descent) ...
1
vote
2answers
72 views

Why do we double the number in a quadratic cost function or MSE?

$$ C(w,b) = \frac{1}{2n}\sum_{x}||y(x)-a||^2 $$ Where y is a 10-dimensional vector, a is the output, w is the weight and b is the bias and n is the number of inputs. If this is the MSE, shouldn't it ...
4
votes
2answers
1k views

What are the cases where it is fine to initialize all weights to zero

I've taken a few online courses in machine learning, and in general, the advice has been to choose random weights for a neural network to ensure that your neurons don't all learn the same thing, ...
2
votes
0answers
311 views

Cost/loss functions for multi-tasking regression neural networks

The mean square loss function is the standard for regression neural networks. However, if I have a neural network learning two tasks (two outputs) at once, is it more advisable to train on the sum of ...
1
vote
1answer
380 views

asymmetric cost function for deep neural network binary classifier

I am building a deep neural network based binary classifier, with single output. The loss function I actually want to minimize is $$ \mathcal L(\hat y,y) = \begin{cases} 0, & \text{if $\hat y$ = ...
2
votes
2answers
626 views

logistic regression algorithm fails to work

I'm trying to code my own logistic regression algorithm using Andrew NG's machine learning using Octave. lectures. So what I did was make a csv file, the first row being some parameter and the second ...
0
votes
1answer
180 views

Derivation of the cross-entropy equation in Michael Nielsen's book

I am reading the book http://neuralnetworksanddeeplearning.com/chap3.html by Michael Nielsen. So this is a question mostly for the people familiar with the book and understanding the material. In the ...
1
vote
1answer
811 views

Gradient Descent in logistic regression

Logistic and Linear Regression have different cost functions. But I don't get how the gradient descent in logistic regression is the same as Linear Regression. We get the Gradient Descent formula by ...
2
votes
2answers
1k views

Softmax classifier never allows for 100% probability in LSTM?

When working with LSTM I am using a softmax classifier and a one-hot encoded vector approach. The softmax looks like this: $$S(h_i) = \frac{e^{h_i}}{\sum e^{h_{total}}}$$ notice, LSTM's result is a $...
0
votes
1answer
127 views

Custom c++ LSTM slows down at 0.36 cost is usual?

Am I missing a part of the puzzle? I have implemented LSTM in c++ which steadily decreases in error, but slows down at the certain error value. It also seems to predict most of the characters, but ...
1
vote
0answers
99 views

How to decrease the learning rate only when cost function is stagnant?

I'm training a very complex function in tensorflow. Is there a way to decrease the learning rate only when the cost isin't decreasing very much?
1
vote
1answer
598 views

Logistic Regression : Solving the cross-entropy cost function analytically

Logistic regression cost function is cross-entropy. It is defined as below: This is a convex function. To reach the minimum, scikit-learn provides multiple types of solvers such as : ‘liblinear’ ...
8
votes
1answer
2k views

Cost function for Ordinal Regression using neural networks

What is the best cost function to train a neural network to perform ordinal regression, i.e. to predict a result whose value exists on an arbitrary scale where only the relative ordering between ...
1
vote
1answer
973 views

Understanding Logistic Regression Cost function

Linear Regression cost function: $$J(\theta) = \frac{1}{2 m} \sum_{i=1}^m (h_{\theta}(x^{(i)}) - y^{(i)})^2$$ where: $$h_{\theta}(x) = \theta_0 + \theta_1 x_1$$ Logistic Regression cost function $...
17
votes
2answers
18k views

Python implementation of cost function in logistic regression: why dot multiplication in one expression but element-wise multiplication in another

I have a very basic question which relates to Python, numpy and multiplication of matrices in the setting of logistic regression. First, let me apologise for not using math notation. I am confused ...
1
vote
1answer
750 views

Is cross-entropy a good cost function if I'm interested in the probabilities of a sample belonging to a certain class?

I'm training a neural network that, for each of six classes, tries to predict the probability that a sample belongs to it. After that, I want to use these probabilities as fractions of the sample ...
0
votes
1answer
92 views

Any Neural Network implementations that allow for a cost function of more than just the network output?

I have an application of a straightforward MLP, for which the cost function is a function of both the network output, in addition to another value calculated from the network weights (actually the ...
1
vote
2answers
6k views

XGBoost change loss function

I'm using XGBoost (through the sklearn API) and I'm trying to do a binary classification. False Positives are much worse for me than False Negatives, how can I take this into account? The API ...
2
votes
2answers
2k views

Does MLP always find local minimum

In linear regression we use the following cost function which is a convex function: We Use the following cost function in ...
1
vote
1answer
876 views

Cost functions penalizing certain types of misclassification more heavily in tensorflow

I have 2 classes of data A and C. I want to create a NN with 3 output nodes, classifying data as either A, B, or C. Classifying an A or C correctly should have zero cost (very good). Classifying ...
0
votes
1answer
117 views

Computing weights in batch gradient descent

I have a reasonably large set of images that I want to classify using a neural network. I can't fil them all into memory at once, so I decided to process them in batches of 200. I'm using an cross-...
1
vote
0answers
193 views

Altered priors for classification trees

I am using the rpart function (in the "rpart" package) in R to fit a cost-sensitive classification tree. I have read the corresponding vignette, which says that misclassication costs are incorporated ...
1
vote
1answer
1k views

Cost of greater than 1, is there an error?

I'm computing cost in the following way: cross_entropy = tf.nn.softmax_cross_entropy_with_logits(y, y_) cost = tf.reduce_mean(cross_entropy); For the first ...