# Understanding SVM mathematics

I was referring SVM section of Andrew Ng's course notes for Stanford CS229 Machine Learning course. On pages 14 and 15, he says:

Consider the picture below:

How can we find the value of $$\gamma^{(i)}$$? Well, $$w/\Vert w\Vert$$ is a unit-length vector pointing in the same direction as $$w$$. Since, point $$A$$ represents $$x^{(i)}$$, we therefore find that the point $$B$$ is given by $$x^{(i)} − \gamma^{(i)}·w/\Vert w\Vert$$. But this point lies on the decision boundary, and all points $$x$$ on the decision boundary satisfy the equation $$w^Tx + b = 0$$. Hence, $$w^T\left(x^{(i)}-\gamma^{(i)}\frac{w}{\Vert w \Vert}\right)+b=0$$ Solving for $$\gamma^{(i)}$$ yields $$\color{red}{\gamma^{(i)}=\frac{w^Tx^{(i)}+b}{\Vert w\Vert}}$$

I am not getting how the last red-colored equality is arrived. I am getting something like this: $$w^T\left(x^{(i)}-\gamma^{(i)}\frac{w}{\Vert w \Vert}\right)+b=0$$ $$\rightarrow w^Tx^{(i)}-\gamma^{(i)}\frac{w^Tw}{\Vert w \Vert}+b=0$$ $$\rightarrow w^Tx^{(i)}+b=\gamma^{(i)}\frac{w^Tw}{\Vert w \Vert}$$

How can I proceed further to equality in red color? Do I have to divide both the sides again by $$\Vert w \Vert$$ to get the following? $$\rightarrow \frac{w^Tx^{(i)}+b}{\Vert w \Vert}=\gamma^{(i)}\frac{w^Tw}{\Vert w \Vert\Vert w \Vert}$$

But then how $$\frac{w^Tw}{\Vert w \Vert\Vert w \Vert}$$ equals to $$1$$?

• That's a pretty standard fact; see (what is currently) the fourth displayed equation at en.wikipedia.org/wiki/Dot_product#Geometric_definition Sep 16, 2021 at 14:23
• Yeah... I myself jotted down little proof long back, but forgot it for a while. Now wondering if $\Vert w\Vert$ is pure magnitude, then how it involves direction component of $w$ and $w^T$ and thus how dividing $w^Tw$ (which involves both direction and magnitude) with $\Vert w\Vert\Vert w\Vert$ (which is pure magnitude) yields $1$ (which is pure magnitude). Am I thinking non-sense?
– Rnj
Sep 16, 2021 at 16:20
• As you see in your linked proof, after the transpose-and-multiply, the result of $w^T w$ is a scalar, not a vector anymore. Sep 16, 2021 at 16:23
• Aahhh, thats again a basic fact: "vector dot product is always scalar"?
– Rnj
Sep 16, 2021 at 16:57

Hint: $$w^Tw = \Vert w \Vert^2$$ this stems directly from the definitions of norm and matrix product (assuming $$w$$ is column vector as usually taken) and one can expand the two sides to prove it easily.
Note that technically $$w^Tw$$ is a $$1 \times 1$$ matrix but any such matrix is identified with its single scalar entry. So it is simply a scalar number. Or equivalently any scalar value is also a $$1 \times 1$$ matrix.