given the following dataset $S = \{(0,1),(1,1),(1,2)\}$ and the regularized problem $$\sum_{i=1}^3 (y_i - w_1 x_i - w_0)^2 + \lambda w_1^2 \quad \lambda = 1 $$ i was tasked with finding the optimal $w_0 ,w_1$ that solve this problem and was advised to use the closed form formula, which i dug from my notes: $$\mathbf{\Theta}^* = \operatorname{argmin}_\theta \frac{1}{N} \|X\Theta -y\|_2^2 + \lambda \Theta^TW\Theta \implies \Theta =(X^TX+N\lambda W)^{-1} X^T y$$ I've managed to compute it but the results don't seem to match the correct ones in the text, or the ones my friends got using partial derivatives. I got $w_0^* = 17/5$ and $w_1^* =1/5$. attached is my calculation, here are the matrices I've defined: $$X = \left[ \begin{array}{cc} 1 & 0 \\1& 1 \\ 1 & 2 \end{array}\right] \quad A = \left[ \begin{array}{cc} 0 & 0 \\0& 1 \end{array}\right] \quad y = \left[ \begin{array}{c} 1 \\1\\ 2 \end{array}\right]$$(A is my stand in for W the weights matrix due to the notation overlap in this particular problem, my W is the formula's Theta - the weights vector).
my attempt: https://i.stack.imgur.com/fviQh.jpg


1 Answer 1


Found an error but it doesn’t solve my problem, Lambda should've been renormalized to 1/3 since the current configuration of the problem didn't have 1/N in front of it EDIT- this doesn't solve my problem! now the solutions are 1 and 1/3 (found a calculation error) - this doesn't match the derivative method OR the official possibly wrong results! The odd thing is, taking A to be a zeros matrix which is what I accidentally did after setting lambda =1/3, yielded the same results my friends got - why is that? enter image description here

  • $\begingroup$ Wouldn't your X just be (0,1,2) and y = (1,1,2)? I'm having a hard time understanding where your A comes from? From the notation W, Theta appear at least to me to refer to the same matrix. Also, what is your lambda? $\endgroup$ May 10, 2023 at 13:31
  • $\begingroup$ redefined the X matrix to include features - look at first pic. I redefined A as theta, and took the formula verbaitim from my course (didn’t change notation) - caused some serious notational overlap. my lambda was 1, redefined it as 1/3 to normalize. talked to a TA and he said the formula only works for l_2 ridge regression and I’d have to see what changes when i change A -> ideally it should vanish. $\endgroup$
    – kal_elk122
    May 11, 2023 at 12:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.