1
$\begingroup$

I have the following question - is it possible to train a model based on Transformer architecture to use additional attributes to generate a caption for an image?

For example, I have a dataset with the following fields: painting description, painting author, genre, date of creation, historical era, etc. I want to generate a caption for the picture not only the text from the description of the picture, but also other attributes.

I was thinking of creating separate classifiers for the attributes, but maybe there is a way to use one model for this purpose.

$\endgroup$

1 Answer 1

1
$\begingroup$

For simplicity in getting something running, you can allocate sections of the token sequence for a transformer into designated sections for your various fields. Then the masked language modelling will handle the learning of those tasks jointly together for you, as long as every sample has the additional features you want, or you are fine trying it out with a "missing token" when the fields are missing.

By allocate, I mean break up the N-length token sequence typically for natural language, sacrificing the token sequence length of the text for the other attributes.

This avoids having to deal with multi-task training of multiple classifiers by using the exact same setup, just changing how the data is fed to the model.

I have seen this approach used in research literature, specifically in attempting to leverage the data sample's extra features or attributes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.