0
$\begingroup$

I want to convert images frame into text, for instance, if I have a image with a dog that plays with a ball I will produce a simple text "Dog plays with ball" (Ok, I know I must have x samples training, but I mean ipotetically).

Which type of NN allows me to perform that action?

Thank you

$\endgroup$

1 Answer 1

2
$\begingroup$

Welcome to SE:DataScience.

What you are looking for is called image captioning. A common approach for this is called an encoder-decoder model, where the encoder is a CNN-based NN to encode the content of an image into a vector representation, and a RNN-based decoder generates its caption conditioned on that vector.

This is a relatively well-studied area, with a lot of well-formatted dataset and open-source implementations, for example in TensorFlow. The link above can also serve as a good tutorial for this topic.

More recent models can use the attention mechanism to focus on specific part (object) in an image, do dense captioning, or answer questions about images, etc.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.