Following Andrew Ng's machine learning course, he explains SVM kernels by manually selecting 3 landmarks and defining 3 gaussian function based on them. Then he says that we are actually defining 3 new features which are $f_1$, $f_2$ and $f_3$.

$\hskip0.9in$enter image description here

And by applying these 3 gaussian functions on every input data: $$x=(x_1,x_2)\to \hat{x}=(f_1(x), f_2(x), f_3(x))$$

it seems that we are mapping our data from $\mathbb R^2$ space to a $\mathbb R^3$ space. Now our goal is to find a hyperplane in the 3 dimensional space, where our transformed data is linearly separable. Is my understanding correct? If not, how these 3 new features should be interpreted?

$\hskip1in$enter image description here

In some blog posts, i have read that by using a gaussian kernel, we are mapping our data to an infinite dimensional space (where gaussian kernel computes the dot product of transformed input data) which contradicts with my above understandings.


Yep, this is the correct interpretation. The kernels make a difficult classification problem into a much simpler one by making the data linearly separable by transforming it into a higher dimension. I think this image does a good job of illustrating that. enter image description here

| improve this answer | |
  • $\begingroup$ So how this interpretation is correct when it's said that gaussian kernels map data to an infinite dimensional space? $\endgroup$ – Mehran Torki Apr 23 at 19:30
  • $\begingroup$ Yes, the kernel maps to infinite space and then also projects it to finite n-dimensional space. This is a pretty good and concise post: shapeofdata.wordpress.com/2013/07/23/gaussian-kernels $\endgroup$ – LiamFiddler Apr 23 at 19:52
  • $\begingroup$ Thanks, i read the post, but i'm not sure i understand. By using a gaussian kernel $K$, we are implicitly mapping our $\mathbb R^2$ input space to an infinite dimensional space (using a mapping function $\phi(x)$ that $K(x,y)=\phi(x) \cdot \phi(y)$). Then by selecting 3 landmarks $l^{(1)}$, $l^{(2)}$, $l^{(3)}$, we are actually selecting 3 dimensions of that infinite dimensional space. Is that right? Are those dimensions $f_1(x)$, $f_2(x)$ and $f_3(x)$ ? $\endgroup$ – Mehran Torki Apr 24 at 20:52
  • $\begingroup$ Essentially, yes. We calculate the distance to each landmark, take the Gaussian function of those distances, and the result gives the new coordinates in the N-Dimensional space. $\endgroup$ – LiamFiddler Apr 27 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.