# Finding optimal weights for models

I'm trying to implement an algorithm to find the minimal value of a function.

Before moving to sigmoid activation functions, i'm trying to understand linear regression.

Usually, a gradient descent algorithm is used to find an minimal value where the algorithm converges, but there are some other ways for linear models.

Say I have two vectors:

x=[1,2,3,4,5,6,7,8,9,10,11,12]

y=[2.3,2.33,2.29,2.3,2.36,2.4,2.46,2.5,2.48,2.43,2.38,2.35]


Between these points, I would like to add a linear separator with least squares.

Say I have some imperfect linear function:

$$f(x)=0.026x+2.3$$

As I know, there are two ways to find this:

$$w = (X^TX)^{-1}X^Ty$$

$$w^{new} = w^{old} - \nu \frac{dy}{dx}$$

Although for linear models finding derivative is trivial, thus second method is not necessary.

Now i've used the first equation on the vectors, in Python:

w = ((np.transpose(x)*x)**-1)*np.transpose(x)*y


Unfortunately, the output was irrelevant:

[ 2.3, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]


Then i've tried using the second method for 500 iterations, in Python:

for i in range(1,5000):
x_old = x_new
x_new = x_old - v*dydx
print("x_new = {0} - {1}({2}) = {3}".format(x_old, v, dydx, x_new))


However, i'm not sure how to know when it reaches a convergence point.

How can I use these methods properly for linear models? And if so, how can they be used for more complex models such as logistic regression?

• Can you define what is perfect? Like in what sense? Mar 9, 2018 at 17:15
• @aditya Perfect for separating classes with least possible squares. Mar 9, 2018 at 17:20
• Nothing as such exists in ML(except SOTA)....If something is performing well for your problem, then that's just perfect  Mar 9, 2018 at 17:22
• Well then that's the definition for ML, but as i know "perfect" for single layer linear separators would be to separate them with least squares. Mar 9, 2018 at 17:24
• @ShellRox what do you mean Although for linear models finding derivative is trivial? Nov 12, 2023 at 15:55

In this case your feature matrix $X$ has a single dimension. Each point in your graph has a $y$ value that depends only on 1 value of $x$.

Ok let's go through the code

x=[1,2,3,4,5,6,7,8,9,10,11,12]
y=[2.3,2.33,2.29,2.3,2.36,2.4,2.46,2.5,2.48,2.43,2.38,2.35]


Let's convert these to matrices. We will also add a column of 1's to the end of the $X$ matrix. This will be used to train the bias value.

temp = np.ones((len(x), 2))
temp[:,0] = np.asarray(x)
x = temp
y = np.asarray(y)


Now we will calculate our weights as

$w = (X^TX)^{-1}X^Ty$

w = np.matmul(np.matmul(np.linalg.inv(np.matmul(np.transpose(x), x)), np.transpose(x)), y)


array([ 0.01174825, 2.30530303])

Look at the dimensions of our weights vector. It only has 2 values. One value associated with $x$, the first column of our $X$ matrix, and a bias, associated with the 1's column that we added. The equation of this line is described as

$y = 0.01174825 x_1 + 2.30530303$

We can see that this line indeed describes the data pretty well for a linear regression.

# Deeper

However, your data looks like it would be better fit using a polynomial. You should try

$y = w_1 x_1^2 + w_2 x_1 + b$

To do this add a new feature in the $X$ matrix which corresponds to $x^2$.

x=[1,2,3,4,5,6,7,8,9,10,11,12]
y=[2.3,2.33,2.29,2.3,2.36,2.4,2.46,2.5,2.48,2.43,2.38,2.35]
temp = np.ones((len(x), 3))
temp[:,0] = np.power(np.asarray(x), 2)
temp[:,1] = np.asarray(x)
x = temp
y = np.asarray(y)

w = np.matmul(np.matmul(np.linalg.inv(np.matmul(np.transpose(x), x)), np.transpose(x)), y)

[![xx = range(1,15,1)
yy = $0$*len(xx)
for ix, i in enumerate(xx):
yy$ix$ = w$0$*i**2 + w$1$*i + w$2$][2]][2]


# Even deeper

And going one step further by adding the $x^3$ term in the same way we get

Make sure not to add too high of a degree to your polynomial or you will be overfitting!! This means although you characterize your training data perfectly, it will not generalize well to new instances. Thus this will be a useless model. That is why you need to split your training and testing data, that way you can verify if the model you build using your training data can generalize.

• Thank you! As i understand the matrix equation only works for linear models correct? Also thank you for mentioning polynomial examples. Mar 9, 2018 at 17:34
• Yeah, each different type of model has its own means of training. Mar 9, 2018 at 17:41
• @JahKnows how do you know it would be better using a polynomial? What's the intuition behind it? Nov 12, 2023 at 12:32

Also you have misunderstood the -1.. It's not the exponent rather the symbol of inverse of a Matrix

Also use .T for Transpose...(a bit Pythonic Convenience)

Just to help you out, Search np.linalg.inv

Just to add a little, you stop when you see that your loss isn't improving any more as such or is improving at the 5th decimal place..