1
$\begingroup$

I would like to know if it is possible to combine rnn and cnn. I explain you :

I have pictures of bikes, cars and moto and every pictures is linked to a text. For instance for a car I can have the following text :

"4 wheels"

For a bike I can have that :

"2 wheels"

or

"handlebar"

And I wonder if I can create a convolutionnal neural network for the pictures and a recurrent neural network for the text. Do you know where I can find informations about that ?

Thank you very much !

PS : when I say every pictures is linked to a text. I don't mean that the text is on the picture but I have two files one file which is a picture and the others a text file.

$\endgroup$
6
  • $\begingroup$ in principle it is possible to combine CNN and RNN yes $\endgroup$
    – Nikos M.
    Jun 28 at 8:49
  • 3
    $\begingroup$ This task has already been treated by many researchers, you should try googling something like 'image captionning deep learning' to find some tutorials / examples of architecture that are used for this kind of task. Here is an example of an AI that basically does your task. $\endgroup$
    – Ubikuity
    Jun 28 at 9:55
  • 2
    $\begingroup$ What is exactly the task you are addressing? Do you want to predict the text from the picture? Do you want to use both the picture and the text to predict a label? $\endgroup$
    – noe
    Jun 28 at 13:35
  • $\begingroup$ Yes, image captioning is the right answer for your combination between RNN and CNN. See this reference as well [arxiv.org/pdf/1612.01033.pdf] (Areas of Attention for Image Captioning). $\endgroup$
    – user119783
    Jun 28 at 22:23
  • $\begingroup$ I think you don't understand what I want to do. Actually, I have pictures and I don't want to describe the picture using text, I want to do predictions using BOTH pictures and texts. Image captionning does not allow that. $\endgroup$
    – Adam
    Jun 30 at 9:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.