3
$\begingroup$

Lets say I am creating a fully convolutional classifier for identifying different types of animals. Now lets assume that I know what part of the animal kingdom the animal belongs to beforehand (mammal, reptile, amphibian .. etc). How would I present this extra information to the fully convolutional network?

Any ideas or research would be very helpful.

Edit* More specifically I am concerned with domain adaptation as shown here, In this example I am using a fully convolutional network without any pooling or striding, where the output is the same size as the input (i.e. an image transformed from one domain to another). In this respect I want to be able run the domain adaptation on images of any size. Adding extra channels to the input seems to make the most sense, however if I am one-hot encoding a bunch of categorical variables, this will consume a lot of memory.

$\endgroup$
1
  • $\begingroup$ Welcome to the site! The simplest way is to add binary features to encode the kingdom. If you want to be more sophisticated, you could learn an embedding for these features to encode phylogenetic similarity, but I imagine the network will do fine on its own. $\endgroup$ – Emre Sep 14 '17 at 18:40
1
$\begingroup$

I've been thinking about this problem before and while I have not researched this or even implemented this, I have a few ideas, from easy/obvious to more elaborate:

  • Don't make it fully convolutional but add some dense layers where you concat this information
  • If it's just one category, one-hot encode it and broadcast it over your picture as extra channels at the input or maybe halfway in the network
  • Similar to the previous suggestion but learn an embedding for your categories that you broadcast either at the beginning or the middle
  • This time, use one-hot encoding as input and learn an embedding that is for example 8x8 and upsample this to your image resolution
  • Learn an embedding again, but this time use some upconvolutions to scale it to the input (or middle) size

Let me know how much it helps!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.