I'm trying to implement an algorithm to find the minimal value of a function.
Before moving to sigmoid activation functions, i'm trying to understand linear regression.
Usually, a gradient descent algorithm is used to find an minimal value where the algorithm converges, but there are some other ways for linear models.
Say I have two vectors:
x=[1,2,3,4,5,6,7,8,9,10,11,12]
y=[2.3,2.33,2.29,2.3,2.36,2.4,2.46,2.5,2.48,2.43,2.38,2.35]
Between these points, I would like to add a linear separator with least squares.
Say I have some imperfect linear function:
$f(x)=0.026x+2.3$
As I know, there are two ways to find this:
$w = (X^TX)^{-1}X^Ty$
and a gradient descent algorithm:
$w^{new} = w^{old} - \nu \frac{dy}{dx}$
Although for linear models finding derivative is trivial, thus second method is not necessary.
Now i've used the first equation on the vectors, in Python:
w = ((np.transpose(x)*x)**-1)*np.transpose(x)*y
Unfortunately, the output was irrelevant:
[ 2.3, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]
Then i've tried using the second method for 500 iterations, in Python:
for i in range(1,5000):
x_old = x_new
x_new = x_old - v*dydx
print("x_new = {0} - {1}({2}) = {3}".format(x_old, v, dydx, x_new))
However, i'm not sure how to know when it reaches a convergence point.
How can I use these methods properly for linear models? And if so, how can they be used for more complex models such as logistic regression?
perfect
? Like in what sense? $\endgroup$