6
$\begingroup$

I have math formula and some data and I need to fit the data to this model. The math is like. $y(x) = ax^k + b$ and I need to estimate the $a$ and the $b$.

I have triet gradient descend to estimate these params but it seems that is somewhat time consuming. Is there any efficient way to estimate the params in a formula?

$\endgroup$
2
  • 1
    $\begingroup$ General-purpose optimization is the general answer to this, either using derivate-free procedures or using gradients. If you can linearize it then, as Dave answered, a linear model is probably best. It does also matter how you formulate the problem, some formulations are much more efficient then others, although they solve the same underlying problem. $\endgroup$ Dec 8, 2022 at 9:02
  • 1
    $\begingroup$ You are asking to fit a model to the data, not the other way around. $\endgroup$
    – Thomas
    Dec 8, 2022 at 12:25

4 Answers 4

12
$\begingroup$

If you know $k$, which it seems you do, then this is just a linear regression. In fact, with just one feature (the $x^k$), this is a simple linear regression, and easy equations apply without you having to resort to matrices.

$$ \hat b=\dfrac{ \text{cov}(x^k, y) }{ \text{var}(x^k) }\\ \hat a =\bar y-\hat b \bar x $$

These are the ordinary least squares estimates of $a$ and $b$.

$\endgroup$
1
  • $\begingroup$ Can I convert this formula and solve it as LinearRegression problem:(I have $x,y$ and I need to find $a,b$) $\LARGE y = \frac{abx}{1+bx} \implies \frac{1}{y} = \frac{1+bx}{abx} \implies \frac{1}{y} = \frac{1}{abx} + \frac{1}{a} \implies \frac{x}{y} = \frac{x}{a} + \frac{1}{ab} $ $\endgroup$ Dec 10, 2022 at 8:26
4
$\begingroup$

The other answer (@Dave) only works in a very specific case, which is basically a linear regression.

In case you want a more general answer, there is subfield in ML called Symbolic regression https://gplearn.readthedocs.io/en/stable/

A symbolic regressor is an estimator that begins by building a population of naive random formulas to represent a relationship. The formulas are represented as tree-like structures with mathematical functions being recursively applied to variables and constants. Each successive generation of programs is then evolved from the one that came before it by selecting the fittest individuals from the population to undergo genetic operations such as crossover, mutation or reproduction.

Or have a look at PySR https://github.com/MilesCranmer/PySR

PySR uses evolutionary algorithms to search for symbolic expressions which optimize a particular objective.

These two methods work in low-dimensional cases.

$\endgroup$
1
  • 1
    $\begingroup$ That is TRUE. Sometimes, we cannot convert the formula to Linear or Polynomial or any known formula which has a close form. That is very helpful. $\endgroup$ Dec 8, 2022 at 5:37
2
$\begingroup$

Just to add to Dave's fine answer, I would like to describe how I would implement the fitting in practice:

In Excel or Google Sheets, I would import my $y$ and $x$ data into two columns. I would then create a third column that is $x*x$, i.e. $x^2$.

If you then plot $y$ vs $x^2$, and fit a line to that data, you would get the $a$ and $b$ variables you are looking for.

$\endgroup$
2
$\begingroup$

It is confusing to me that no one has used the term "nonlinear least squares" to describe the technique commonly used to solve this general kind of problem, finding the parameters which minimize the (squared, summed) difference between a set of data and an equation of a known form with floating parameters, so I am going to say "nonlinear least squares". As Dave has pointed out the nonlinear part is not necessary for this particular problem, but if you had some unpleasant equation (in python) like

def gross_function(x, a, b, c, d):
    return tanh(a*x + b*x**2) + cos(2*pi*x*c + d)

then

params, cov = scipy.optimize.curve_fit(gross_function, xdata, ydata)

will give you the least squares optimized values of the paramaters and the corresponding covariance matrix (or more likely fail to converge if you don't provide an initial guess for a moderately complex equation like this). There are many more optional parameters: initial guess, bounds, minimization method, errors to weight the points by, precomputed jacobian, but you can read the documentation to learn about those.

The default method used for unconstrainted problems is the Levenberg–Marquardt algorithm, which combines the Gauss-Newton algorithm and gradient descent.

$\endgroup$
1
  • $\begingroup$ Great, Thanks... $\endgroup$ Dec 9, 2022 at 9:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.