I was referring SVM section of Andrew Ng's course notes for Stanford CS229 Machine Learning course. On pages 14 and 15, he says:

Consider the picture below:
enter image description here

How can we find the value of $\gamma^{(i)}$? Well, $w/\Vert w\Vert$ is a unit-length vector pointing in the same direction as $w$. Since, point $A$ represents $x^{(i)}$, we therefore find that the point $B$ is given by $x^{(i)} − \gamma^{(i)}·w/\Vert w\Vert$. But this point lies on the decision boundary, and all points $x$ on the decision boundary satisfy the equation $w^Tx + b = 0$. Hence, $$w^T\left(x^{(i)}-\gamma^{(i)}\frac{w}{\Vert w \Vert}\right)+b=0$$ Solving for $\gamma^{(i)}$ yields $$\color{red}{\gamma^{(i)}=\frac{w^Tx^{(i)}+b}{\Vert w\Vert}}$$

I am not getting how the last red-colored equality is arrived. I am getting something like this: $$w^T\left(x^{(i)}-\gamma^{(i)}\frac{w}{\Vert w \Vert}\right)+b=0$$ $$\rightarrow w^Tx^{(i)}-\gamma^{(i)}\frac{w^Tw}{\Vert w \Vert}+b=0$$ $$\rightarrow w^Tx^{(i)}+b=\gamma^{(i)}\frac{w^Tw}{\Vert w \Vert}$$

How can I proceed further to equality in red color? Do I have to divide both the sides again by $\Vert w \Vert$ to get the following? $$\rightarrow \frac{w^Tx^{(i)}+b}{\Vert w \Vert}=\gamma^{(i)}\frac{w^Tw}{\Vert w \Vert\Vert w \Vert}$$

But then how $\frac{w^Tw}{\Vert w \Vert\Vert w \Vert}$ equals to $1$?

  • 1
    $\begingroup$ That's a pretty standard fact; see (what is currently) the fourth displayed equation at en.wikipedia.org/wiki/Dot_product#Geometric_definition $\endgroup$ Sep 16 at 14:23
  • $\begingroup$ Yeah... I myself jotted down little proof long back, but forgot it for a while. Now wondering if $\Vert w\Vert$ is pure magnitude, then how it involves direction component of $w$ and $w^T$ and thus how dividing $w^Tw$ (which involves both direction and magnitude) with $\Vert w\Vert\Vert w\Vert$ (which is pure magnitude) yields $1$ (which is pure magnitude). Am I thinking non-sense? $\endgroup$
    – Rnj
    Sep 16 at 16:20
  • $\begingroup$ As you see in your linked proof, after the transpose-and-multiply, the result of $w^T w$ is a scalar, not a vector anymore. $\endgroup$ Sep 16 at 16:23
  • $\begingroup$ Aahhh, thats again a basic fact: "vector dot product is always scalar"? $\endgroup$
    – Rnj
    Sep 16 at 16:57

Hint: $w^Tw = \Vert w \Vert^2$ this stems directly from the definitions of norm and matrix product (assuming $w$ is column vector as usually taken) and one can expand the two sides to prove it easily.

Note that technically $w^Tw$ is a $1 \times 1$ matrix but any such matrix is identified with its single scalar entry. So it is simply a scalar number. Or equivalently any scalar value is also a $1 \times 1$ matrix.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.