What is the dimension hopping problem in machine learning (occurring in convolutional neural networks and image recognition)? I have googled about it but all I get is information on the Physics of material shape deformation. It will be more helpful to me if some one explain it with an example related to machine learning. Can anyone help me out with this or point me toward resources that can?
$\begingroup$
$\endgroup$
Add a comment
|
5 Answers
$\begingroup$
$\endgroup$
5
Welcome to DataScience.SE! I'd never heard of this problem so I looked it up. It is explained on the third slide of this presentation by Geoff Hinton:
More things that make it hard to recognize objects
• Changes in viewpoint cause changes in images that standard learning methods cannot cope with.
– Information hops between input dimensions (i.e. pixels)
• Imagine a medical database in which the age of a patient sometimes hops to the input dimension that normally codes for weight!
– To apply machine learning we would first want to eliminate this dimension-hopping.
In other words, it is about conceptual features migrating or hopping from one input feature dimension to another while still representing the same thing. One would like to be able to capture or extract the essence of the feature while being invariant to which input dimension it is encoded on.
-
$\begingroup$ I understand that in Computer Vision one wants to be invariant for places in the image, but I don't get the age example. $\endgroup$ Commented May 26, 2016 at 22:10
-
$\begingroup$ I took that to mean age and weight are dependent, but I'm not sure; it's not my presentation! Or maybe they meant they literally use the wrong column and we want to detect that. $\endgroup$– EmreCommented May 26, 2016 at 22:19
-
$\begingroup$ @sdream I only made a comment; Emre gave the answer. (But you should probably still accept it). The point with CNNs is that not only one feature changes when an object is somewhere else, but a complete pattern is at a different input. $\endgroup$ Commented May 27, 2016 at 7:29
-
$\begingroup$ @Emre, what I am getting from your answer is that, no matter from which direction a specific property is input, the feature which this specific property causes should be invariant to input dimension of this property. Thanks! :). Still waiting for some more specific answers, otherwise will mark your reply as answer. $\endgroup$ Commented May 27, 2016 at 7:41
-
3$\begingroup$ The age example is supposed to highlight a dataset that does not have dimension-hopping. Age and weight do not "hop" or swap values randomly between examples - they are not interchangeable and the example is showing how odd that would be (and how difficult it would make simple tasks such as linear regression). Pixel values in images (and similar data in many signal processing tasks) do interchange or move easily due to the nature of the problem. $\endgroup$ Commented May 29, 2016 at 18:01
$\begingroup$
$\endgroup$
As far as I understand the issue is the following: In image recognition the inputs to your network could be the pixels (grayscale or only 1 and 0 for black and white). If you want to, e.g. recognize handwritten numbers it is very difficult to only work with such values as you never know where exactly the number (i.e. the black values) will be.
Is pixel 140 black or 142 black? In both cases it could well be a three. In the age/weight example these inputs are well defined. Feature 2 is weight. Feature 3 is age. These "dimensions" shouldnt "hop" in your dataset.
So: In your picture training the "threes" or "cars" or "houses" have to be recognized independent of their location in the picture, i.e. the pixel values, i.e. the feature/input vector, i.e. the dimensions as opposed to clearly defined inputs such as patient data.
How do you solve this in image recognition? You use additional tricks, e.g. convolution.
$\begingroup$
$\endgroup$
I read the previous answers, and Neil Slater's comment to Emre's post, copied again below, hits the nail. "Dimension hopping" is a term created by Dr. Hinton of machine learning pioneer fame in the context of view point. To quote Dr. Hinton "So, typically envision the input dimensions correspond to pixels, and, if an object moves in the world and you don't move your eyes to follow it, the information about the object will occur on different pixels." Age and weight are input dimension that are not easily confused. Dr. Hinton used this obviously NOT likely dimension hopping situation of age and weight of patients to mean we would certainly be able to spot and fix any erroneous between these types of data (It's hard not to notice most adults are under 100 years old and more than 100 pounds). The likely problem of dimension hopping, which Dr. Hinton was addressing, is pixels could be displaced because we have a different view point (e.g. the object could have moved or we are looking at it from a different angle). Linear neural networks would not be able to detect this, whereas convolutional neural networks by design would.
"The age example is supposed to highlight a dataset that does not have dimension-hopping. Age and weight do not "hop" or swap values randomly between examples - they are not interchangeable and the example is showing how odd that would be (and how difficult it would make simple tasks such as linear regression). Pixel values in images (and similar data in many signal processing tasks) do interchange or move easily due to the nature of the problem. – Neil Slater May 29 '16 at 18:01"
$\begingroup$
$\endgroup$
Explanation straight from Hinton's course on Neural Networks for Machine Learning ....
" Dimension hopping occurs when one can take the information contained in the dimensions of some input, and move this between dimensions while not changing the target. The canonical example is taking an image of a handwritten digit and translating it within the image. The dimensions that contain "ink" are now different (they have been moved to other dimensions), however the label we assign to the digit has not changed. Note that this is not something that happens consistently across the dataset, that is we may have a dataset containing two handwritten digits where one is a translated version of the other, however this still does not change the corresponding label of the digits."
$\begingroup$
$\endgroup$
Hoping is only about issues with portion of image or pixels moving within dimension (mostly) and sometime into other dim (different receptive field) but output remains same.
This issue is dealt with invariance or equivariance and looks like weight and age example is easy way to state. Suppose if we are aware of this weight and age hopping we would easily make changes to the algo and get right result. But like data/information hopping, image hopping also happens, if we considered a '4' and a '4' shifted several pixels to the left to be different classes which has different target.
With Translation Invariance or better equivariance throguh filter this movement or hopping is not much issue though it increases complexity and at the cost of throwing away information, such as location.
Pls let me know if you need more clarity I will try to.
