One is: $$J=-\frac{1}{m}\sum_{i=1}^{m}\sum_{k=1}^{K}\Big[y_{k}^{i}\log\big((h_{\theta}(x^{i}))_k\big)+(1-y_{k}^{i})\log\big(1-(h_{\theta}(x^{i}))_k\big)\Big]$$

The other one is: $$J=-\frac{1}{m}\sum_{i=1}^{m}\Big[y^{i}\log(a^{i})+(1-y^{i})\log(1-a^{i})\Big]$$

As I can see those two equations are not equal. How both can be used to calculate cost function?

Also, one of them using $h$ function which is $a$ of output layer, whereas others are using $a$ ($a$ is $f(w*x)$ where $f$ is activation function). When I looked from the book "Pattern Recognition and Machine Learning" from Bishop, he used $a$ for both of the equations. But from another source which I took equations from $h$ is used. But using different $a$ values and using just one of them (namely $h$ which is $a$ of output) are totally different things.

Both sources are reliable, what am I missing?

  • $\begingroup$ You aren't missing anything imho, that's just different way to write eqns. (I believe one is vectorized and other one is having double sigma(looping over individual items)) will get back to this. $\endgroup$ – Aditya Jul 15 '19 at 2:53
  • $\begingroup$ I didn't give much attention because I usually use vectorized version but what is $$h(x^{i})$$ if $$h(x)=(theta*a^{l-1})$$ where l is total number of layers? what it becomes equal to? $\endgroup$ – J.Smith Jul 15 '19 at 3:20
  • $\begingroup$ Are you sure that the second sum in the first formula is supposed to be $\sum_{k=1}^k$? $\endgroup$ – Elias Strehle Jul 15 '19 at 13:40
  • $\begingroup$ @EliasStrehle Capital k, not lowercase. Sorry. $\endgroup$ – J.Smith Jul 15 '19 at 15:59

I don't remember exactly what the book has mentioned, but I guess the difference between the two is due to having one or multiple features. I guess it is already mentioned in the book. They are the same. One is for multi-dimensional input, and the other is for one-dimensional. One sigma is iterating over the features and the other is iterating over the examples. You can put $k$ to one to achieve the simpler formula.

  • $\begingroup$ You changed your handler! (Comment will be deleted) $\endgroup$ – Aditya Jul 15 '19 at 5:27
  • $\begingroup$ @Aditya opppsss :D $\endgroup$ – Media Jul 15 '19 at 5:45
  • $\begingroup$ @Vaalizaadeh Yes, that is for two sums. But how about the uses of 'h' and 'a'? If you know the answer can you add it to yours? $\endgroup$ – J.Smith Jul 15 '19 at 15:58
  • $\begingroup$ @J.Smith It is just a naming convention, $h_{\theta}(x)$ = $a$. $a$ is considered as the activation of last layer. $\endgroup$ – Media Jul 16 '19 at 5:14

ideally cost functions are used to identify the global minima(where the error value is least) for all the input values. what so ever the cost function it will only for check for least value, using different methods but value remain same for all cost function.

cost function graph

in the above image the value which is about to touch the x axis that is the best cost values and that remains same in all cost functions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.