How to Precondition A Fully Convolutional Neural Network

Question

Lets say I am creating a fully convolutional classifier for identifying different types of animals. Now lets assume that I know what part of the animal kingdom the animal belongs to beforehand (mammal, reptile, amphibian .. etc). How would I present this extra information to the fully convolutional network?

Any ideas or research would be very helpful.

Edit* More specifically I am concerned with domain adaptation as shown here, In this example I am using a fully convolutional network without any pooling or striding, where the output is the same size as the input (i.e. an image transformed from one domain to another). In this respect I want to be able run the domain adaptation on images of any size. Adding extra channels to the input seems to make the most sense, however if I am one-hot encoding a bunch of categorical variables, this will consume a lot of memory.

Answer 1

I've been thinking about this problem before and while I have not researched this or even implemented this, I have a few ideas, from easy/obvious to more elaborate:

• Don't make it fully convolutional but add some dense layers where you concat this information
• If it's just one category, one-hot encode it and broadcast it over your picture as extra channels at the input or maybe halfway in the network
• Similar to the previous suggestion but learn an embedding for your categories that you broadcast either at the beginning or the middle
• This time, use one-hot encoding as input and learn an embedding that is for example 8x8 and upsample this to your image resolution
• Learn an embedding again, but this time use some upconvolutions to scale it to the input (or middle) size

Let me know how much it helps!