I'm trying to implement an algorithm to find the minimal value of a function.

Before moving to sigmoid activation functions, i'm trying to understand linear regression.

Usually, a gradient descent algorithm is used to find an minimal value where the algorithm converges, but there are some other ways for linear models.

Say I have two vectors:



enter image description here

Between these points, I would like to add a linear separator with least squares.

Say I have some imperfect linear function:


As I know, there are two ways to find this:

$w = (X^TX)^{-1}X^Ty$

and a gradient descent algorithm:

$w^{new} = w^{old} - \nu \frac{dy}{dx}$

Although for linear models finding derivative is trivial, thus second method is not necessary.

Now i've used the first equation on the vectors, in Python:

w = ((np.transpose(x)*x)**-1)*np.transpose(x)*y

Unfortunately, the output was irrelevant:

[ 2.3, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]

Then i've tried using the second method for 500 iterations, in Python:

for i in range(1,5000):
    x_old = x_new
    x_new = x_old - v*dydx
    print("x_new = {0} - {1}({2}) = {3}".format(x_old, v, dydx, x_new))

However, i'm not sure how to know when it reaches a convergence point.

How can I use these methods properly for linear models? And if so, how can they be used for more complex models such as logistic regression?

  • $\begingroup$ Can you define what is perfect? Like in what sense? $\endgroup$
    – Aditya
    Mar 9, 2018 at 17:15
  • $\begingroup$ @aditya Perfect for separating classes with least possible squares. $\endgroup$
    – ShellRox
    Mar 9, 2018 at 17:20
  • $\begingroup$ Nothing as such exists in ML(except SOTA)....If something is performing well for your problem, then that's` just perfect ` $\endgroup$
    – Aditya
    Mar 9, 2018 at 17:22
  • $\begingroup$ Well then that's the definition for ML, but as i know "perfect" for single layer linear separators would be to separate them with least squares. $\endgroup$
    – ShellRox
    Mar 9, 2018 at 17:24
  • $\begingroup$ @ShellRox what do you mean Although for linear models finding derivative is trivial? $\endgroup$ Nov 12, 2023 at 15:55

2 Answers 2


In this case your feature matrix $X$ has a single dimension. Each point in your graph has a $y$ value that depends only on 1 value of $x$.

Ok let's go through the code


Let's convert these to matrices. We will also add a column of 1's to the end of the $X$ matrix. This will be used to train the bias value.

temp = np.ones((len(x), 2))
temp[:,0] = np.asarray(x)
x = temp
y = np.asarray(y)

Now we will calculate our weights as

$w = (X^TX)^{-1}X^Ty$

w = np.matmul(np.matmul(np.linalg.inv(np.matmul(np.transpose(x), x)), np.transpose(x)), y)

array([ 0.01174825, 2.30530303])

Look at the dimensions of our weights vector. It only has 2 values. One value associated with $x$, the first column of our $X$ matrix, and a bias, associated with the 1's column that we added. The equation of this line is described as

$y = 0.01174825 x_1 + 2.30530303$

enter image description here

We can see that this line indeed describes the data pretty well for a linear regression.


However, your data looks like it would be better fit using a polynomial. You should try

$y = w_1 x_1^2 + w_2 x_1 + b$

To do this add a new feature in the $X$ matrix which corresponds to $x^2$.

temp = np.ones((len(x), 3))
temp[:,0] = np.power(np.asarray(x), 2)
temp[:,1] = np.asarray(x)
x = temp
y = np.asarray(y)

w = np.matmul(np.matmul(np.linalg.inv(np.matmul(np.transpose(x), x)), np.transpose(x)), y)

[![xx = range(1,15,1)
yy = \[0\]*len(xx)
for ix, i in enumerate(xx):
    yy\[ix\] = w\[0\]*i**2 + w\[1\]*i + w\[2\]][2]][2]

enter image description here

Even deeper

And going one step further by adding the $x^3$ term in the same way we get

enter image description here

Make sure not to add too high of a degree to your polynomial or you will be overfitting!! This means although you characterize your training data perfectly, it will not generalize well to new instances. Thus this will be a useless model. That is why you need to split your training and testing data, that way you can verify if the model you build using your training data can generalize.

  • $\begingroup$ Thank you! As i understand the matrix equation only works for linear models correct? Also thank you for mentioning polynomial examples. $\endgroup$
    – ShellRox
    Mar 9, 2018 at 17:34
  • $\begingroup$ Yeah, each different type of model has its own means of training. $\endgroup$
    – JahKnows
    Mar 9, 2018 at 17:41
  • $\begingroup$ @JahKnows how do you know it would be better using a polynomial? What's the intuition behind it? $\endgroup$ Nov 12, 2023 at 12:32

Also you have misunderstood the -1.. It's not the exponent rather the symbol of inverse of a Matrix

Also use .T for Transpose...(a bit Pythonic Convenience)

Just to help you out, Search np.linalg.inv

For your second Query,

Refer here

Just to add a little, you stop when you see that your loss isn't improving any more as such or is improving at the 5th decimal place..

Hope this helps..


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.