Solving multi label image classification using TimeDistributed dense layer

Question

I have a multi label image dataset having 5 labels. Each image can have more than one label at the same time. I am using a convolutional neural network to extract features and those extracted features I am giving to RepeatVector layer to create 5 copies of extracted features and after RepeatVector layer I have connected TimeDistributed layer with TimeDistributed(Dense(2)).

y_train is a 3D array and its shape is (1600, 5, 2) and x_train is an array of images.

For example:

>>> x_train.shape
(1600, 3, 100, 100)

>>> y_train.shape 
(1600,5,2)

>>> y_train[0] = 
array([[0, 1],   # [0,1] = 1 label present and  [1,0] = 0 label abset
       [1, 0],
       [1, 0],
       [1, 0],
       [1, 0]])

Code:

def get_label(y):
  tmp = []
  d = {0:[1,0],1:[0,1]}   # 0 absent 1 present
  for i,value in enumerate(y):
    tmp.append( d[value] )
  return tmp


X,Y= get_data()
Y = Y.tolist()
y = []
for value in Y:
  y.append(get_label(value))

Y = np.array(y,dtype=int)


x_train, x_test, y_train, y_test = train_test_split(X,Y,test_size =0.2,random_state=100)


img_channels = 3
img_rows     = 100
img_cols     = 100
nb_classes   = 5 

model = Sequential()

model.add(Convolution2D(32, 3, 3, border_mode='same',input_shape=(img_channels, img_rows, img_cols)))
model.add(Activation('relu'))
model.add(Convolution2D(32, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Convolution2D(64, 3, 3, border_mode='same'))
model.add(Activation('relu'))
model.add(Convolution2D(64, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Flatten())
model.add(Dense(512))
model.add(Activation('relu'))
model.add(RepeatVector(nb_classes))
model.add(TimeDistributed(Dense(2)))
model.add(Activation('softmax'))

# let's train the model using SGD + momentum (how original).
# opt = RMSprop(lr=0.001, rho=0.9, epsilon=1e-06)
opt = SGD(lr=0.01, momentum=0.0, decay=1e-6, nesterov=False)
model.compile(loss='categorical_crossentropy', optimizer=opt,metrics=['accuracy'])
model.fit(x_train,y_train,nb_epoch=10,batch_size=32,validation_data=(x_test,y_test),shuffle=True)
out = model.predict_classes(x_test)

But after training, I get all zeros for the test set. Is this approach wrong?

Answer 1

There is no need to repeat any features. What you want to do is change your activation function from softmax to sigmoid, otherwise leaving everything else as it would normally be done for binary classification.

...
model = Sequential()

model.add(Convolution2D(32, 3, 3, border_mode='same',input_shape=(img_channels, img_rows, img_cols)))
model.add(Activation('relu'))
model.add(Convolution2D(32, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Convolution2D(64, 3, 3, border_mode='same'))
model.add(Activation('relu'))
model.add(Convolution2D(64, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Flatten())
model.add(Dense(512))
model.add(Activation('relu'))
model.add(Dense(5)) # whatever the number of classes is
model.add(Activation('sigmoid'))
...

You will then need to simplify your Y tensor from N x 5 x 2 to N x 5, e.g. with Y = Y[:, :, 0].

This may or may not be why you get all zeros, but it is certainly a more standard approach to a multi-label classification than using RepeatVector and TimeDistributed. Something else to look at is how much you are training, 5 epochs with only 1600 examples is not a lot. Similarly, a Dense(512) is very large for your small dataset size. Maybe reduce that down to something like Dense(16)?