24
$\begingroup$

I'm training a convolutional neural network to classify images on fog conditions (3 classes). However, for each of about 150.000 images I also have four meteorological variables available that might help in predicting the classes of the images. I was wondering how I could add the meteorological variables (e.g. temperature, wind speed) to the existing CNN structure so that it can help in the classification.

One way I can already think of is creating another (small) feedforward neural net alongside the CNN and then concatenating the outputs of the CNN layers and the hidden layers of the non-image neural net to each other at the dense layer.

The second way I could think of is just contacting these features to the dense layer. However, in this case, the non-image variables will (I think) only be able to make linear predictions.

Are there any other (better) ways that the non-image features could be included in the model? And what would be the advisable method considering the amount of data I have?

Another question I have is whether or not I should unfreeze the convolutional layers while training with these non-image features? These layers of a Resnet-18 (which were initialized as pre-trained on ImageNet) have already been fine-tuned using the images. My guess is that I should keep them frozen and only unfreeze the dense layer since it is only here that the non-image features come into 'contact' with the image features (not earlier in the CNN). If I'm wrong on this, please say so!

Improve this question
$\endgroup$
4
  • $\begingroup$ you could model the join distribution between additional_features and images via some generative model like GAN, VAE. then you could get the latent variables and use it with a supervised criterion $\endgroup$
    – Fadi Bakoura
    Commented May 8, 2018 at 15:42
  • $\begingroup$ I'm in a similar situation myself. I'm using a stack of sky images for the past 15 minutes to try and predict the output of solar panels close to the camera 15 minutes into the future. I recently decided to bring several weather features into play (one for each image like in your case). Your first suggestion worked much better than the second one (of directly appending non-graphical features to the dense layer). To be precise, the second suggestion led to issues with normalization. I found that for some reason I can't explain yet, the Batchnorm layer was not able to normalize the graphical fea $\endgroup$
    – Vignesh Venugopal
    Commented Jun 19, 2018 at 22:19
  • $\begingroup$ @VigneshVenugopal please mention me in the comments otherwise I can't be notified. What's your question? :) $\endgroup$
    – Green Falcon
    Commented Jul 9, 2018 at 12:04
  • $\begingroup$ How can I concatenate speed &throttle & steering angle to my networks? Would you please explain about dense how many dense add ? What is depends on it ? $\endgroup$
    – Nasrinzaghari
    Commented Sep 28, 2019 at 5:37

1 Answer 1

Reset to default
16
$\begingroup$

My solution is like your first recommendation, but with slight changes.

  1. Construct your convolutional layers and stack them till the flatten-layer. This network should be fed with the image data.
  2. Flat your activation maps
  3. Construct a fully connected network with the desired number of neurons and layers.
  4. Concatenate the outputs of the flattened layer of the convolutional net and the fully connected net.
  5. add some dense layers and connect them to the last layer which represents your classes.

You can use customary cost functions for this architecture.

Improve this answer
$\endgroup$
4
  • $\begingroup$ In Keras, can you do it with Concatenate() layers? $\endgroup$
    – Leevo
    Commented Jul 4, 2019 at 12:57
  • $\begingroup$ Yes. You should concatenate them to put them alongside each other. $\endgroup$
    – Green Falcon
    Commented Jul 4, 2019 at 13:11
  • $\begingroup$ Shall I use Concatenate() or concatenate() layers? I can't tell the difference $\endgroup$
    – Leevo
    Commented Jul 4, 2019 at 13:12
  • $\begingroup$ It depends on the way you want to create your network. By the way, you can see the argument list of each. They differ. You may also want to take a look at here. $\endgroup$
    – Green Falcon
    Commented Jul 4, 2019 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.