1
$\begingroup$

I'm trying to understand can we implement a simple linear regression model.

Let's say we are predicting price currencies. We want to know whether the currency will raise or not.

As i understand, we need to define two vectors for this:

$x=[1,2,3,4,5,6,7,8,9,10,11,12]$ - months

$y=[2.30,2.33,2.29,2.30,2.36,2.40,2.46,2.50,2.48, 2.43,2.38,2.35]$ - average prices.

Let's plot this:

enter image description here

Before sigmoid separator, i will try a simple linear separator.

I'm guessing that at first, I need to choose some random slope (and bias will be the smallest scalar in vector).

$f(x) = 0.026x+2.3$

enter image description here

As we see, the separator is inaccurate, let's try the quadratic cost function at f(1):

$C = \frac{1}{N} \sum_{i=0}^{N}(\hat{y} - y)^2$

$C = \frac{1}{1} \sum_{i=0}^{1}(2.3259999999999996-2.30)^2=0.0006759999999999897$

It seems accurate in the beginning, but it gets worse as it progresses, so somehow i need to improve it.

From my knowledge, the next step is to find the derivative of the function.

Normally, gradient descent algorithm is used for this, but finding a slope of the tangent line is very easy here:

$\frac{dy}{dx} = 0.026$

What is the next step? How can i use this derivative to use proper weights to minimize the cost function?

Improve this question
$\endgroup$
4
  • $\begingroup$ datascience.stackexchange.com/questions/28747/… $\endgroup$
    – JahKnows
    Commented Mar 8, 2018 at 17:00
  • 1
    $\begingroup$ $w^{new} = w - \nu dy/dx$ $\endgroup$
    – JahKnows
    Commented Mar 8, 2018 at 17:02
  • $\begingroup$ @JahKnows Apologies, i'm confused again, isn't gradient descent algorithm used to estimate the derivative of the function? (i'm assuming that is the equation used in iteration of gradient descent algorithm) $\endgroup$
    – ShellRox
    Commented Mar 8, 2018 at 17:05
  • $\begingroup$ Gradient descent is a tool for finding the variables which minimize a function. However, as you noted this is not necessary for minimizing the equation of a line. We have a closed form solution. $\endgroup$
    – JahKnows
    Commented Mar 8, 2018 at 17:08

1 Answer 1

Reset to default
2
$\begingroup$

You are correct for a linear separator line the derivative seems trivial. In this case gradient descent is not necessary because a closed form solution exists for the weights.

$w = (X^TX)^{-1}X^Ty$

We only use optimization techniques such as gradient descent for models where a closed form solution does not exist. However, if you perform gradient descent on the weights using the derivative and doing the following equation iteratively

$w^{new} = w^{old} - \nu \frac{dy}{dx}$

would yield the same result as the closed form solution. This is however very unnecessary when not necessary.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Apologies if i'm mistaken but derivative is not used in the equation? Is this different for other models? $\endgroup$
    – ShellRox
    Commented Mar 8, 2018 at 17:16
  • 1
    $\begingroup$ The derivative is just a part of the gradient descent algorithm. By following the slope of a function (the derivative) we can identify where the base of the mountain is. Thus we have found the minimum of the function. There are other optimization algorithms which can equally be used to train models. $\endgroup$
    – JahKnows
    Commented Mar 9, 2018 at 1:45
  • $\begingroup$ Also one more question if i may ask, could you explain how does closed form solution work? As i know most of linear regression models give slope and bias (which are weighted at perfect point) and then the operation is performed on whole array. But when i performed the closed form solution on vectors X (months) and y (price) the output seemed irrelevant. Did i do something wrong? Thank you! $\endgroup$
    – ShellRox
    Commented Mar 9, 2018 at 10:26
  • 1
    $\begingroup$ For the bias term you will want to concatenate your X matrix with a column of 0's. This will add the needed bias. If you have 2 features, then you will end up with two weights associated with those features and a bias term if you use the above equation. You can post a question with an example of your dataset and I can walk through how to set up a linear model. $\endgroup$
    – JahKnows
    Commented Mar 9, 2018 at 10:32
  • $\begingroup$ I've used x and y vectors above ($x=[1,2,3,4,5,6,7,8,9,10,11,12]$, $y=[2.3, 2.33, 2.29, 2.3, 2.36, 2.4, 2.46, 2.5, 2.48, 2.43, 2.38, 2.35]$) and then tested it for equation above in Python's numpy: w = ((np.transpose(x)*x)**-1)*np.transpose(x)*y, the output: [ 2.3, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]. $\endgroup$
    – ShellRox
    Commented Mar 9, 2018 at 11:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.