Multi task learning architecture for Multi-label classification

Question

I am working a classification problem. The dataset was collected from Painters by number, a competition hosted by kaggle. The task is to identify painter,style and genre given paintings.

So far, I trained individual models to predict painter,style,genre given paintings. Now i would like to incorporate Multi task learning (i.e) Developing a single model which can predict all three tasks.

The issue i am currently facing is in designing architecture.

      Model                         No of classes (Softmax)
   (Individual Models)                   
   --------------------             -----------------------
 Model predicts painter                     8 (8 - painters)
 given paintings 

 Model predicts style                       10 (10 style classes)
 given paintings      


 Model predicts genre                       23 (23 genre classes)
 given paintings    

I don't know how to combine the above models in keras. Any suggestions or feedback would be helpful.

Answer 1

You should design a multi-task model (MTM). MTM has the ability to share learned representations from input between several tasks. More precisely, we try to simultaneously optimize a model with m types of loss function, one for each task. Consequently, MTM will learn more generic features, which should be used for several tasks, at its earlier layers. Then, subsequent layers, which become progressively more specific to the details of the desired task, can be divided into multiple branches, each for a specific task.

You need an architecture like the following:

# Your input and hidden layers
inp = Input(...)     
x = Layer1(...)(inp)
x = Layer2(...)(x)
...
x = Layer_N-1(...)(x) 

# Your 'm' output layers
out_1 = Layer_N(x) 
out_2 = Layer_N(x)
...
out_m = Layer_N(x)

# your model
MTM = Model(inputs=inp, outputs=[out_1,out_2,...,out_m])
MTM.compile(loss=[loss_func_1, loss_func_2,..., loss_func_m], 
      optimizer='xxx', 
      metrics=xxx)

MTM.fit(train_data, [train_labels_1,train_labels_2,...,train_labels_m],                
          batch_size = xxx,
          epochs = xxx)

If you want a real example in Keras, I have implemented a 2-task ConvNet for my project here.