4
$\begingroup$

I am working with linear regression and I would like to know the Time complexity in big-O notation. The cost function of linear regression without an optimisation algorithm (such as Gradient descent) needs to be computed over iterations of the weight combinations (as a brute force approach). This makes computation time dependent on the number of weights and obviously on the number of training data.

If $n$ is the number of training data, $W$ is the number of weights and each resolution of the weight space is set to $m$ meaning that each weight will iterate through $m$ number of possible values. Then the time complexity of this linear regression is

$O(m^Wn)$

Is this correct?

$\endgroup$
  • $\begingroup$ ai.stackexchange.com/questions/5728/… $\endgroup$ – DuttaA Jul 20 '18 at 20:02
  • $\begingroup$ @DuttaA how is this relevant? $\endgroup$ – user134132523 Jul 22 '18 at 15:27
  • $\begingroup$ Sorry I read with an optimisation algo...I didn't know you were trying by matrix methods $\endgroup$ – DuttaA Jul 22 '18 at 15:52
1
$\begingroup$

It highly depends on the "solver" you are using.

Calling $n$ the number of observations and $p$ the number of weights, the overall complexity should be $n^2p+p^3$.

Indeed, when performing a linear regression you are doing matrices multiplication whose complexity is $n^2p$ (when evaluating $X'X$) and inverting the resulting matrix. It is now a square matrix with $p$ rows, the complexity for matrix inversion usually is $p^3$ (though it can be lowered).

Hence a theoretical complexity : $n^2p+p^3$.

Side notes

However, numerical simulations (using python's scikit library) seem to have a time complexity close to $n^{0.72} p^{1.3}$

This may be due to the fact that no implementation actually perform the full inversion (instead, the system can be solved using gradient descents) or due to the fact that there are other ways to calibrate the weights of a linear regression.

Source

An article from my blog : computational complexity of machine learning algorithms

| improve this answer | |
$\endgroup$
  • $\begingroup$ why is the exponent of n, squared. if I have 3 weights then the error function computed over n training sets have to be recomputed for all the possible combinations of those 3 weights right. same goes for higher number of weights. that is why my reasoning was that the number of weights should be an exponent. and also in your calculation where is the limit on the number of possible iterations per weight? I am talking about a purely brute force approach even though its not the optimal way to go about it. $\endgroup$ – user134132523 Sep 3 '18 at 18:01
  • $\begingroup$ "why is the exponent of n, squared. if I have 3 weights then the error function computed over n training sets have to be recomputed for all the possible combinations of those 3 weights right." No. This would be true if you were doing a naive approach, but based on the closed formula for $\hat{\beta}$, you do not have to try every combination of weights $\endgroup$ – RUser4512 Sep 4 '18 at 8:09
  • $\begingroup$ from : math.stackexchange.com/questions/84495/…, you may have inverted the exponent on n and p. The complexity should be $np^2$, this is also in line with practical time complexity. $\endgroup$ – lcrmorin Apr 15 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.