Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I've been watching Dr. G. Hinton lectures on Neural Networks in Machine Learning, and in one of the lectures he explains what the goals of Unsupervised Learning are.
I am having trouble understanding the part where high-dimensional inputs such as images live on or near a low-dimensional manifold (or several such manifolds). What is a manifold exactly, and why is this the case?
Thanks!
Depends who you ask but generally mannifold is just some structure in a (high) dimensional space having finite dimension : a line, a curve, a plane, a rock, a sphere, a ball, a cylinder, a torus, a "blob"... something like this :
If you ask a mathematician they would say that it is a general term describing "a curve" (dimension 1) or "surface" (dimension 2), or a 3D object (dimension 3)... for any possible finite dimension $n$. A one dimensional manifold is simply a curve (line, circle...). A two dimensional manifold is simply a surface (plane, sphere, torus, cylinder...). A three dimensional manifold is a "full object" (ball, full cube, the 3D space around us...).
TO ANSWER YOUR QUESTION what he is referring is that information in this high-dimensional space can be compressed and perserved in some smaller space. (remember images when quantified live in some space that contains information about them)
A manifold is some kind of low-dimensions structure that exists in a higher-dimensional space. The classic example of this is the Swiss Roll dataset, which simply looks like a spiral with values that vary monotonically along the curves (represented by colors here).
.
The overall idea is that there is a simple, 1-dimensional representation of the color values, which can be seen if you unfurl the roll. But if you look in the original space and try to do some kind of nearest-neighbor clustering, you might determine that the dark blue and yellow-green samples are most similar, since they are closest in the original space. Finding the low-dimensional manifold allows you to cluster only along the manifold axis (which is a spiral in this case), giving you the "proper" representation of which samples are close to one another.
