I designed a neural network model with large number of output predicted by softmax function. However, I want categorize all the outputs into 5 outputs without modifying the architecture of other layers. The model performs well in the first case but when I decrease the number of output it loses accuracy and get a bad generalization. My question is: Is there a method to make my model performs well even if there is just 5 outputs? for example: adding dropout layer before output layer, using other activation function, etc.


2 Answers 2


I believe it should work if we update the last Softmax units and update the weight of the last layer accordingly.

I have tried this with MNIST digit and seems to work

#Let's assume "model" is the trained model on 10 outputs

saved_model = model.save('/content/model.h5')
model_reload = keras.models.load_model('/content/model.h5')

#Create model_2 with 5 output
from keras.models import Sequential
model_2 = tf.keras.Sequential()
for layer in model_reload.layers[:-1]: # go through until last layer
model_2.add(keras.layers.Dense(5, activation='softmax'))

#Get the last layer weight, this is where you have to take care while slicing the weight matrix depending on which 5 Class you are retaining
#We are taking digits (0, 2, 4, 6, 8)
for layer in model_reload.layers:
    if 'Dense' in str(layer):
        if layer.units==10:
            w=layer.get_weights()[0][:,::2] ##Note this slicing weight
            b=layer.get_weights()[1][::2] ##Note this slicing of bias

dataset_new = dataset_new[(dataset_new.label % 2 = 0)] #Only even digits
x = dataset_new.iloc[:,1:]
y = dataset_new.iloc[:,0]

###Normalize the data
x = x/255

y_pred = model_2.predict_classes(x)
y_pred = y_pred * 2 #Model will predict 0-4(This multiply is too map it to 0,2,4,6,8

from sklearn.metrics import accuracy_score

0.9514600845439969, It is better than the base accuracy

You may check the same in this Notebook


First make sure your dataset is labelled properly in 5 distinct classes from 0 to 4 and no more. Then add dropout layer in between each layer with prob of 0.1 and gradually increase till 0.5 until you find a good val score. In your optimizer add weight decay term with value around 0.1. Find some appropriate regularizer for your data type online.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.