# Regression problem - too complex for gradient descent

I try to predict temperatures values as function of time and different parameters. The temperature curve look like a "ramp" with some "gauss peaks" on regular intervals.

So, I try to build a regression model for the following equation:

$$\hat{T}_{a, \mu, \sigma, s, c, m, p}\left(t\right) = a \frac{1}{\sigma\sqrt{2 \pi}} e^{-\frac{\left(\sin\left(\frac{t-s}{c}\right) - \mu\right)^2}{2 \sigma^2}} + m t + p$$

My goal is to establish the values of $a$, $\mu$, $\sigma$, $s$, $c$, $m$, $p$.

My first intention was to use the gradient descent, but I'm pretty sure that the derivative of the RSS based cost function won't be convex, so I might get stuck at local mimina.

I'm also wondering if there is no better alternative to solve that problem. It concerns especially methods not involving minimizing derivatives. I think (but not sure) that neuronal networks can help me with that.

So, in short, how would you solve the following?

$$\left[\begin{array}{l} a \\ \mu \\ \sigma \\ s \\ c \\ m \\ p \end{array}\right] \leftarrow \textrm{argmin}_{a, \mu, \sigma, s, c, m, p}\frac{1}{2N} \sum_{i=1}^N{\left(a \frac{1}{\sigma\sqrt{2 \pi}} e^{-\frac{\left(\sin\left(\frac{t_i-s}{c}\right) - \mu\right)^2}{2 \sigma^2}} + m t + p - T_i\right)^2}$$

Thanks a lot for your help.

P.S.: I'm using python 3 with scipy stack

• The $\frac{1}{2N}$ factor in the cost function is there to make usage of derivatives simpler... Commented Dec 15, 2015 at 13:29
• Neural networks are, for the most part, are trained using gradient descent. Although your problem may not be convex, using gradient descent with a good type of regularization should give you a "good enough" solution. Using global optimizers, such as genetic algorithms, does not guarantee a global optimum will be found, and it will be much more computationally expensive than using gradient descent. Are you strictly against using first order optimization? Commented Dec 15, 2015 at 19:42
• Thanks Armen. No, I'm not against first order optimizations ... for the good reason that I don't know what first order means ;-) Commented Dec 15, 2015 at 22:07
• By first order optimization, I mean optimizations that take into account the first derivative. Just a fancy way of saying gradient descent :) Commented Dec 15, 2015 at 22:59
• Thanks. Good to know. I think I'll use gradient descent. Commented Dec 16, 2015 at 8:38

## 1 Answer

A loss function that is non-convex can still minimized with gradient descent. Even if the loss function converges to a local mimina, those could be a useful set of parameters.

If you choose to not use gradient descent, scipy has several methods to optimization functions. Try as many as applicable and see which method requires the best set of parameters.