Accuracy drops if more layers trainable - weird

Question

I'm training a 4-class classifier. I load pre-trained weights from vgg16, and only set the last layer to be trainable. This gives me a loss (categorical crossentropy) of 0.9 and a training accuracy of 70%.

However, I want features for my images - the penultimate activations is one choice for getting features for every image. Hence I set the last and the penultimate layers to both be trainable, but then my loss jumps to 9.8 and training accuracy drops to 39%.

How can this happen? Here's the code I'm using:

vgg.model.pop()
for layer in vgg.model.layers:
    layer.trainable = False
vgg.model.add(Dense(trn_batches.nb_class, activation='softmax'))
opt = Adam(lr=0.001)

#only the last layer is trainable
vgg.model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
vgg.model.fit_generator(trn_batches, samples_per_epoch=trn_batches.nb_sample, nb_epoch=1, 
    validation_data=val_batches, nb_val_samples=val_batches.nb_sample)

#make last 2 layers trainable (starting from -3 because there is a dropout layer)
for layer in vgg.model.layers[-3:]:
    layer.trainable = True
vgg.model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
vgg.model.fit_generator(trn_batches, samples_per_epoch=trn_batches.nb_sample, nb_epoch=1,   validation_data=val_batches, nb_val_samples=val_batches.nb_sample)

Answer 1

Accuracy is not a function of number of features. You might have heard about feature selection processes, and that is the reason we use feature selection or dimensionality reduction. There is huge theory behind this, but in very generic terms, we need to find out the best features, to make our model least complex, still keeping all relevant features in the model. One example, can be adding a new feature reduces accuracy, which is highly biased towards one out come and hence it creates inaccuracy in your results.

This may also happen, if all your features are not normalised, and hence the newly added features is biasing your results and hence introducing inaccuracy.

Answer 2

You probably need to get more training data as you increase your features to justify the increase. Insufficient data can cause the accuracy to be reduced.