# Accuracy drops if more layers trainable - weird

I'm training a 4-class classifier. I load pre-trained weights from vgg16, and only set the last layer to be trainable. This gives me a loss (categorical crossentropy) of 0.9 and a training accuracy of 70%.

However, I want features for my images - the penultimate activations is one choice for getting features for every image. Hence I set the last and the penultimate layers to both be trainable, but then my loss jumps to 9.8 and training accuracy drops to 39%.

How can this happen? Here's the code I'm using:

vgg.model.pop()
for layer in vgg.model.layers:
layer.trainable = False

#only the last layer is trainable
vgg.model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
vgg.model.fit_generator(trn_batches, samples_per_epoch=trn_batches.nb_sample, nb_epoch=1,
validation_data=val_batches, nb_val_samples=val_batches.nb_sample)

#make last 2 layers trainable (starting from -3 because there is a dropout layer)
for layer in vgg.model.layers[-3:]:
layer.trainable = True
vgg.model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
vgg.model.fit_generator(trn_batches, samples_per_epoch=trn_batches.nb_sample, nb_epoch=1,   validation_data=val_batches, nb_val_samples=val_batches.nb_sample)