Is it possible to implement mutiple softmaxes in the last layer in Keras? So the sum of Nodes 1-4 = 1; 5-8 = 1; etc.
Should I go for a different network design?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
8
I would use the functional interface.
Something like this:
from keras.layers import Activation, Input, Dense
from keras.models import Model
from keras.layers.merge import Concatenate
input_ = Input(shape=input_shape)
x = input_
x1 = Dense(4, x)
x2 = Dense(4, x)
x3 = Dense(4, x)
x1 = Activation('softmax')(x1)
x2 = Activation('softmax')(x2)
x3 = Activation('softmax')(x3)
x = Concatenate([x1, x2, x3])
model = Model(inputs=input_, outputs=x)
answered Oct 10, 2017 at 20:27
-
1$\begingroup$ Worth noting the cost function is also going to require similar custom work. $\endgroup$ Oct 10, 2017 at 20:57
-
3$\begingroup$ I'm not suggesting you add the solution, but I think leaving answer as-is gives the impression that OP's model-building work would be complete. But there's an equal extra amount of work for OP to do in other parts of code if they want to actually train the model. You could at least reference that requirement. Same applies to other answer . . . $\endgroup$ Oct 11, 2017 at 6:58
-
2$\begingroup$ @NeilSlater you are absolutely right. I have no idea why I need a different cost function. Can you tell me why this is important? $\endgroup$ Oct 11, 2017 at 10:45
-
1$\begingroup$ @arthurDent - because Keras' multi-class cross-entropy loss is probably not geared up to cope with three simultaneous true classes on each example, and the separation into groups - error in one group may result in gradients incorrectly assigned to outputs in other groups. You could just try it and see what happens . . . it may still converge, but the balance point might not be as good as having three entirely separate networks. $\endgroup$ Oct 11, 2017 at 11:42
-
1$\begingroup$ @arthurDent: . . . although I'm thinking it through in more detail, and the gradient at the logit for multiclass cross entropy with softmax, which is simply $\mathbf{\hat{y}} - \mathbf{y}$ may still apply and work successfully. A normal softmax output would of course fail to learn 3 classes simultaneously, but perhaps this answer is all you need after all . . . $\endgroup$ Oct 11, 2017 at 11:49
$\begingroup$
$\endgroup$
0
It is possible just implement your own softmax function. You can split a tensor to parts, then compute softmax separately per part and concatenate tensor parts:
def custom_softmax(t):
sh = K.shape(t)
partial_sm = []
for i in range(sh[1] // 4):
partial_sm.append(K.softmax(t[:, i*4:(i+1)*4]))
return K.concatenate(partial_sm)
concatenate without axis argument concatenate through last axis (in our case axis=1).
Then you can include this activation function in a hidden layer or add it to a graph.
Dense(activation=custom_activation)
or
model.add(Activation(custom_activation))
You also need to define a new cost function.