given the following dataset $S = \{(0,1),(1,1),(1,2)\}$ and the regularized problem
$$\sum_{i=1}^3 (y_i - w_1 x_i - w_0)^2 + \lambda w_1^2 \quad \lambda = 1 $$ i was tasked with finding the optimal $w_0 ,w_1$ that solve this problem and was advised to use the closed form formula, which i dug from my notes: $$\mathbf{\Theta}^* = \operatorname{argmin}_\theta \frac{1}{N} \|X\Theta -y\|_2^2 + \lambda \Theta^TW\Theta \implies \Theta =(X^TX+N\lambda W)^{-1} X^T y$$
I've managed to compute it but the results don't seem to match the correct ones in the text, or the ones my friends got using partial derivatives. I got $w_0^* = 17/5$ and $w_1^* =1/5$. attached is my calculation, here are the matrices I've defined:
$$X = \left[ \begin{array}{cc} 1 & 0 \\1& 1 \\ 1 & 2 \end{array}\right] \quad A = \left[ \begin{array}{cc} 0 & 0 \\0& 1 \end{array}\right] \quad y = \left[ \begin{array}{c} 1 \\1\\ 2 \end{array}\right]$$(A is my stand in for W the weights matrix due to the notation overlap in this particular problem, my W is the formula's Theta - the weights vector).
my attempt:
https://i.sstatic.net/fviQh.jpg
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Found an error but it doesn’t solve my problem, Lambda should've been renormalized to 1/3 since the current configuration of the problem didn't have 1/N in front of it
EDIT- this doesn't solve my problem! now the solutions are 1 and 1/3 (found a calculation error) - this doesn't match the derivative method OR the official possibly wrong results! The odd thing is, taking A to be a zeros matrix which is what I accidentally did after setting lambda =1/3, yielded the same results my friends got - why is that?
-
$\begingroup$ Wouldn't your X just be (0,1,2) and y = (1,1,2)? I'm having a hard time understanding where your A comes from? From the notation W, Theta appear at least to me to refer to the same matrix. Also, what is your lambda? $\endgroup$ Commented May 10, 2023 at 13:31
-
$\begingroup$ redefined the X matrix to include features - look at first pic. I redefined A as theta, and took the formula verbaitim from my course (didn’t change notation) - caused some serious notational overlap. my lambda was 1, redefined it as 1/3 to normalize. talked to a TA and he said the formula only works for l_2 ridge regression and I’d have to see what changes when i change A -> ideally it should vanish. $\endgroup$ Commented May 11, 2023 at 12:16