# In SVM Algorithm, why vector w is orthogonal to the separating hyperplane?

I am a beginner on Machine Learning. In SVM, the separating hyperplane is defined as $y = w^T x + b$. Why we say vector $w$ orthogonal to the separating hyperplane?

• An answer to a similar question (for neural networks) is here. – bogatron Jun 9 '15 at 16:39
• @bogatron - I agree with you completely. But my ones just a SVM specific answer. – untitledprogrammer Jun 10 '15 at 19:43
• Except it isn't. Your answer is correct but there is nothing about it that is specific to SVMs (nor should there be). $w^{T}x=b$ is simply a vector equation that defines a hyperplane. – bogatron Jun 10 '15 at 22:01

Geometrically, the vector w is directed orthogonal to the line defined by $w^{T} x = b$. This can be understood as follows:

First take $b = 0$. Now it is clear that all vectors, $x$, with vanishing inner product with $w$ satisfy this equation, i.e. all vectors orthogonal to w satisfy this equation.

Now translate the hyperplane away from the origin over a vector a. The equation for the plane now becomes: $(x − a)^{T} w = 0$, i.e. we find that for the offset $b = a^{T} w$, which is the projection of the vector $a$ onto the vector $w$.

Without loss of generality we may thus choose a perpendicular to the plane, in which case the length $\vert\vert a \vert\vert = \vert b \vert /\vert\vert w\vert\vert$ which represents the shortest, orthogonal distance between the origin and the hyperplane.

Hence the vector $w$ is said to be orthogonal to the separating hyperplane.

The reason why $w$ is normal to the hyper-plane is because we define it to be that way:

Suppose that we have a (hyper)plane in 3d space. Let $P_0$ be a point on this plane i.e. $P_0 = x_0, y_0, z_0$. Therefore the vector from the origin $(0,0,0)$ to this point is just $<x_0,y_0,z_0>$. Suppose that we have an arbitrary point $P (x,y,z)$ on the plane. The vector joining $P$ and $P_0$ is then given by: $$\vec{P} - \vec{P_0} = <x-x_0, y-y_0, z-z_0>$$ Note that this vector lies in the plane.

Now let $\hat{n}$ be the normal (orthogonal) vector to the plane. Therefore: $$\hat{n} \bullet (\vec{P}-\vec{P_0}) = 0$$ Therefore: $$\hat{n} \bullet \vec{P}- \hat{n} \bullet \vec{P_0} = 0$$ Note that $-\hat{n} \bullet \vec{P_0}$ is just a number and is equal to $b$ in our case, whereas $\hat{n}$ is just $w$ and $\vec{P}$ is $x$. So by definition, $w$ is orthogonal to the hyperplane.

Let the decision boundary be defined as $$w^Tx + b = 0$$. Consider the points $$x_a$$ and $$x_b$$, which lie on the decision boundary. This gives us two equations:

$$\begin{equation} w^Tx_a + b = 0 \\ w^Tx_b + b = 0 \end{equation}$$

Subtracting these two equations gives us $$w^T.(x_a - x_b) = 0$$. Note that the vector $$x_a - x_b$$ lies on the decision boundary, and it is directed from $$x_b$$ to $$x_a$$. Since the dot product $$w^T.(x_a - x_b)$$ is zero, $$w^T$$ must be orthogonal to $$x_a - x_b$$, and in turn, to the decision boundary.

Using the algebraic definition of a vector being orthogonal to a hyperplane:

$\forall \ x_1, x_2$ on the separating hyperplane,

$$w^T(x_1-x_2)=(w^Tx_1 + b)-(w^Tx_2 + b)=0-0=0 \ \small\Box.$$