6
$\begingroup$

On several tutorials on object and pattern recognition in images (such as the typical mnist digit recognition problem), when working with a sample of $m$ images of $n \times n$ pixels each, the first step is to build a matrix of $m \times (n * n)$, where each image is converted into a single row-vector with all of its pixels, one next to the other. Different techniques are then applied to this matrix, such as PCA.

My question is: when converting a two-dimensional image to a single-dimensional vector by placing each image row next to another, the fact of which pixels are neighbours (in the same column, different row) seems to be lost. For instance, in image:

a b c
d e f
g h i

Which is converted to:

a b c d e f g h i

The fact that b and e were next to each other seems to be lost.

Is that conclusion correct? Does that fact even matter for analysis? Are there other ways to represent the image that do preserve that piece of information?

Apologies in advance if the question is too vague or basic, I'll be glad to reformulate if needed. Thank you.

$\endgroup$
  • 1
    $\begingroup$ A really nice question :) Updated it with the appropriate tag. Welcome to the site! $\endgroup$ – Dawny33 Jan 27 '16 at 16:02
8
$\begingroup$

The short answer is it depends on two things: what definition of information you use and if you really use that information.

From information theory point of view, if your transformation is reversible than the information is there. This happens because you can apply the inverse transformation to recover the original. So nothing is lost. This is similar with what happens in various lose-less compression algorithms.

So, information is there, but is encoded in another place, specifically in the transformation function. As a consequence, to learn also the position you have to create those new features to express that information. Not also that when you have a classical learner this information is not understood by learner. Take for example a linear model. There is no difference between $f_1 = \beta_0 +\beta_1 x_1 + \beta_2 x_2$ and $f_2 = \beta_2 x_2 + \beta_0 + \beta_1 x_1$. So learning with a simple linear model is impossible to use directly this locality information. This happens with most learners. So, even in the original space the locality is an information in your head, and not in the data.

On the other hand if your neural net is tailored for images and expects to receive patches of images where the order of the values is important than yes, your neural net perhaps understands those relations and use them. Even so, perhaps that custom neural net has a custom way to specify the location information, so what you will have to do would be to encode this information in a format specific to that custom neural net.

On the other hand if we are talking about a standard back propagation neural net, the order is irrelevant. The model is simply not able to use the location.

As a conclusion, if you have a standard generic learning algorithm your only your only chance would be to encode this locality information somehow in the features. For example (this is a made-up example, I do not even know if that would word) one can add a new set of input variable, one for each square of 4 adjacent pixels and use the average of pixels intensity as value. You do not made that explicitly enough, but you created a new set of features which is created using the locality information.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.