# How can you include information not present in an image for neural networks?

I am training a CNN to identify objects in images (one label per image). However, I have additional information about these images that cannot be retrieved by looking at the image itself. In more detail, I'm talking about the physical location of this object. This information proved to be important when classifying these objects.

However, I can't think of a good solution to include this information in a image recognition model, since the CNN is classifying the object based on the pixel values and not on ordered feature data.

One possible solution I was thinking of was to have an additional simple ML model on tabular data (including mainly the location data), such as an SVM, to give give a certain additional weight to the output of the CNN. Would this be a good strategy? I can't seem to find anything in the literature about this.

edit: Someone asked what I meant by 'location'. With the location I meant the physical location of where the image was taken, in context of a large 2d space. I don't want to go too deep into the domain, but it's basically an (x,y) vector on a surface area, and obviously this meta-data cannot be extracted by looking at the pixel values.

edit2: I want to propose an additional way I found that was useful, but was not mentioned in any answer. Instead of using the neural network to predict the classes, I use the neural network to produce features.

I removed the final layer, which resulted in an output of shape 1024x1. This obviously depends on the design of your network. Then, I can use these features together with the meta-data (in my case location data) in an additional model to make predictions, such an SVM or another NN.

• @D.W.I edited the question, thanks. – seb Feb 24 '20 at 15:24

Other answers suggest to put an additional channel, I disagree. I think it's a very computationally intensive, time consuming process. Moreover, it forces non-pixel data to be processed by Conv filters, which doesn't make much sense IMHO.

I suggest you to establish a multi-input model. It would be composed by three parts:

• A Convolutional part, to process pixel data,
• A Feed-forward part to process non-image data,
• Another Feed-forward part that elaborates the prediction based on the concatenation of the two outputs above.

You will need to instantiate them separately, then combine together in a Keras Model(). You will also need Concatenate() layers to combine the two different sources of data.

You can read more about the implementation of multi-input Neural Networks here.

• Note that this way the convnet has no access to the extra information and cannot learn to process the image in different ways depending on the extra information. Whether that's reasonable or not depends on the exact problem. – isarandi Feb 22 '20 at 17:59
• Yes, I admit I'm fine with that. If the Net is deep enough, different parts of it will specialize at different things successfully. Image and non-image data would interact anyway later in the model. – Leevo Feb 23 '20 at 9:24

Edit: after the edit in the question, 1) does not relate so much anymore, but 2) still does.

1. It depends a bit on the form of the location data. If you have a segmentation mask (i.e. another image with two colors denoting for each pixel if it belongs to the object or not), then going with another channel as n1k31t4 suggested might be a good idea.

2. If you have the coordinates or something in a vector form, Figure 2 in this paper shows a way to put the information together. Essentially, the authors concatenate the additional info (in your case the location data) to the output of the feature extractor and feed that into the classifier of the CNN.

• Given the edit, I think this is answering a different question than what was asked. The location is not a location within the image. – D.W. Feb 24 '20 at 16:56
• Thanks for pointing it out. I edited the answer. – matthiaw91 Feb 24 '20 at 21:26

The simplest thing to try out is to put the information in an extra channel of the image.

So if you have RGB channels, you could add a four channel which would simply be the location information you have, repeated for every pixel.

This creates a lot of redundancy, of course, but means you can take any standard image classifier and it will still work.

• Thanks, that makes sense. Will it not be too biased towards that extra channel then, since they all have the same value? In my case it's only grey values, so half of the data will then consist of pixel values, and the other half of location data. Or is a deep neural net with enough epochs robust enough for this? – seb Feb 21 '20 at 9:40
• It has worked in my experience, but as I mentioned, it is the simplest way and perhaps not the most computationally efficient. I suppose you could argue about the efficacy for the model's learning. If the (gray) images change at a macro-level for you separate locations, it could be a gelpful feature for the convolutions. Otherwise there are likely more efficient ways, such feeding the location as a separate I out and concatenating it with some of the last activation maps of the conv model. You just need to experiment a bit. – n1k31t4 Feb 21 '20 at 12:56

I will definitely try one more approach in this case, which is explained below.

1. Will use simple CNN architecture, followed by fully connected layers.
2. Say, now I have fully connected layer(FL) of size 100.
3. Using this FL apply another liner regression model(followed by activation layer...). Structure of linear regression would be:

y = w1i.FL(Ni)+ w2i.f1+ w3i.f2...and so on.

Ni = ith Neuron

f1, f2.. are the non image feature.

w1i, w2i, w3i.. are the weights

What I'm trying to achieve is, using the output of each FL neuron, I will create a linear regression model, where other features would be your non-image data.

Here my assumption is during training, it will boost the weights of neurons based upon the non-image data as well.

In the specific case of knowing the location of the object in the image, one technique would be to crop and pad each training example so that the object is in the exact center. This way the extra information is passed to the neural network implicitly. This is how most face identification neural networks work.

If the "location" of the object is more abstract, like "bedroom" or "Spain," then I'd recommend concatenating the information to each pixel of the image. Don't be afraid to add a large number of extra input channels, neural networks handle this well. For example, Alpha Go has a 48 channel input layer.