Finding optimal weights for models

Question

I'm trying to implement an algorithm to find the minimal value of a function.

Before moving to sigmoid activation functions, i'm trying to understand linear regression.

Usually, a gradient descent algorithm is used to find an minimal value where the algorithm converges, but there are some other ways for linear models.

Say I have two vectors:

x=[1,2,3,4,5,6,7,8,9,10,11,12]

y=[2.3,2.33,2.29,2.3,2.36,2.4,2.46,2.5,2.48,2.43,2.38,2.35]

Between these points, I would like to add a linear separator with least squares.

Say I have some imperfect linear function:

$f(x)=0.026x+2.3$

As I know, there are two ways to find this:

$w = (X^TX)^{-1}X^Ty$

and a gradient descent algorithm:

$w^{new} = w^{old} - \nu \frac{dy}{dx}$

Although for linear models finding derivative is trivial, thus second method is not necessary.

Now i've used the first equation on the vectors, in Python:

w = ((np.transpose(x)*x)**-1)*np.transpose(x)*y

Unfortunately, the output was irrelevant:

[ 2.3, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]

Then i've tried using the second method for 500 iterations, in Python:

for i in range(1,5000):
    x_old = x_new
    x_new = x_old - v*dydx
    print("x_new = {0} - {1}({2}) = {3}".format(x_old, v, dydx, x_new))

However, i'm not sure how to know when it reaches a convergence point.

How can I use these methods properly for linear models? And if so, how can they be used for more complex models such as logistic regression?

Answer 1

In this case your feature matrix $X$ has a single dimension. Each point in your graph has a $y$ value that depends only on 1 value of $x$.

Ok let's go through the code

x=[1,2,3,4,5,6,7,8,9,10,11,12]
y=[2.3,2.33,2.29,2.3,2.36,2.4,2.46,2.5,2.48,2.43,2.38,2.35]  

Let's convert these to matrices. We will also add a column of 1's to the end of the $X$ matrix. This will be used to train the bias value.

temp = np.ones((len(x), 2))
temp[:,0] = np.asarray(x)
x = temp
y = np.asarray(y)

Now we will calculate our weights as

$w = (X^TX)^{-1}X^Ty$

w = np.matmul(np.matmul(np.linalg.inv(np.matmul(np.transpose(x), x)), np.transpose(x)), y)

array([ 0.01174825, 2.30530303])

Look at the dimensions of our weights vector. It only has 2 values. One value associated with $x$, the first column of our $X$ matrix, and a bias, associated with the 1's column that we added. The equation of this line is described as

$y = 0.01174825 x_1 + 2.30530303$

We can see that this line indeed describes the data pretty well for a linear regression.

Deeper

However, your data looks like it would be better fit using a polynomial. You should try

$y = w_1 x_1^2 + w_2 x_1 + b$

To do this add a new feature in the $X$ matrix which corresponds to $x^2$.

x=[1,2,3,4,5,6,7,8,9,10,11,12]
y=[2.3,2.33,2.29,2.3,2.36,2.4,2.46,2.5,2.48,2.43,2.38,2.35] 
temp = np.ones((len(x), 3))
temp[:,0] = np.power(np.asarray(x), 2)
temp[:,1] = np.asarray(x)
x = temp
y = np.asarray(y)

w = np.matmul(np.matmul(np.linalg.inv(np.matmul(np.transpose(x), x)), np.transpose(x)), y)

[![xx = range(1,15,1)
yy = \[0\]*len(xx)
for ix, i in enumerate(xx):
    yy\[ix\] = w\[0\]*i**2 + w\[1\]*i + w\[2\]][2]][2]

Even deeper

And going one step further by adding the $x^3$ term in the same way we get

Make sure not to add too high of a degree to your polynomial or you will be overfitting!! This means although you characterize your training data perfectly, it will not generalize well to new instances. Thus this will be a useless model. That is why you need to split your training and testing data, that way you can verify if the model you build using your training data can generalize.

Answer 2

Also you have misunderstood the -1.. It's not the exponent rather the symbol of inverse of a Matrix

Also use .T for Transpose...(a bit Pythonic Convenience)

Just to help you out, Search np.linalg.inv

For your second Query,

Refer here

Just to add a little, you stop when you see that your loss isn't improving any more as such or is improving at the 5th decimal place..

Hope this helps..