# How to add non-image features along side images as the input of CNNs

I'm training a convolutional neural network to classify images on fog conditions (3 classes). However, for each of about 150.000 images I also have four meteorological variables available that might help in predicting the classes of the images. I was wondering how I could add the meteorological variables (e.g. temperature, wind speed) to the existing CNN structure so that it can help in the classification.

One way I can already think of is creating another (small) feedforward neural net alongside the CNN and then concatenating the outputs of the CNN layers and the hidden layers of the non-image neural net to each other at the dense layer.

The second way I could think of is just contacting these features to the dense layer. However, in this case, the non-image variables will (I think) only be able to make linear predictions.

Are there any other (better) ways that the non-image features could be included in the model? And what would be the advisable method considering the amount of data I have?

Another question I have is whether or not I should unfreeze the convolutional layers while training with these non-image features? These layers of a Resnet-18 (which were initialized as pre-trained on ImageNet) have already been fine-tuned using the images. My guess is that I should keep them frozen and only unfreeze the dense layer since it is only here that the non-image features come into 'contact' with the image features (not earlier in the CNN). If I'm wrong on this, please say so!

• you could model the join distribution between additional_features and images via some generative model like GAN, VAE. then you could get the latent variables and use it with a supervised criterion – Fadi Bakoura May 8 '18 at 15:42
• I'm in a similar situation myself. I'm using a stack of sky images for the past 15 minutes to try and predict the output of solar panels close to the camera 15 minutes into the future. I recently decided to bring several weather features into play (one for each image like in your case). Your first suggestion worked much better than the second one (of directly appending non-graphical features to the dense layer). To be precise, the second suggestion led to issues with normalization. I found that for some reason I can't explain yet, the Batchnorm layer was not able to normalize the graphical fea – Vignesh Venugopal Jun 19 '18 at 22:19
• @VigneshVenugopal please mention me in the comments otherwise I can't be notified. What's your question? :) – Media Jul 9 '18 at 12:04
• How can I concatenate speed &throttle & steering angle to my networks? Would you please explain about dense how many dense add ? What is depends on it ? – Nasrinzaghari Sep 28 '19 at 5:37

My solution is like your first recommendation, but with slight changes.

1. Construct your convolutional layers and stack them till the flatten-layer. This network should be fed with the image data.
• In Keras, can you do it with Concatenate() layers? – Leevo Jul 4 '19 at 12:57
• Shall I use Concatenate() or concatenate() layers? I can't tell the difference – Leevo Jul 4 '19 at 13:12