I was curious to see if one can use a cost function on a set of data points to find the "optimial minimum" solution for any given set of data.

I know for a regular set of data that is clustered symmetrically following a straight regression line is easy to find the proper cost function


Linear model

but what if the data makes a funky shape like that of x^2 or 3*x^3 or etc...

As you can see that the the second graph is a lot more noisy than the first.

quadratic model x^2

Then what do you do? is it the same process or is it different? Can you find an optimal Minimum for any set of data points?

I know the cost function is using the residuals to form its best fit but I was just curious if it can do with any set of data.


I don't think the second picture necessarily has more noise, it's just less linear than the first one. If you are doing regression (which is the case in both your problems) you can use the same loss function to optimize your family of functions that you are using to fit your data. The only difference is that with certain function families gradient descent might not get to a global optimum but will get stuck in a local one. The loss function only determines how to penalize certain errors (based on the residuals).

  • $\begingroup$ If you were to run a neural network how could you use gradient descent/ cost function to optimize the second picture? Sorry I’m still learning.... $\endgroup$ Nov 10 '17 at 15:38
  • $\begingroup$ The way you define your neural network determines the family of functions that the network can take. In the case of a quadratic fit you would have a weird neural network if you would only allow quadratic shapes to be there (that is more traditional regression). Normally you would just take a normal neural network with a few layers and feed the (x,y) pairs with a mean squared error loss. Your line would look more wobbly though. $\endgroup$ Nov 11 '17 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.