3
$\begingroup$

With the keras functional api it is possible to write something like this:

x = Activation('relu')(x)
x = Dense(8, activation='softmax')(x)

My question is whether the Activation() function is a separate Layer (equivalent to Dense(128, activation='relu') or if not... why and when this notation is used?

$\endgroup$
5
$\begingroup$

As stated in the docs, the activation layer in keras is equivalent to a dense layer with the same activation passed as an argument.

This would be equivalent

x = Dense(64)(x)
x = Activation('relu')(x)

is equivalent to

x = Dense(8, activation='relu')(x)

As per your example if the activation layer is used as a layer, this will act as a transformation of the outputs of the previous layer.

As you can see in the model

inputs = Input(shape=(784,))
x = Dense(32, activation='tanh')(inputs)
x = Activation('relu')(x)
predictions = Dense(8, activation='softmax')(x)

The output of the first layer is the result of a densely connected layer with a tanh function. Then these outputs will each be transformed by a relu function. You can see how this is simply a transformation and does not introduce any new model parameters using

model.summary()

enter image description here

|improve this answer|||||
$\endgroup$
  • $\begingroup$ In my case multiple Conv1D Layers are concatenated and then the code above is added: x = Activation('relu')(x) x = Dense(8, activation='softmax')(x) .... does this make any sense? Why should one perform Activation before Dense when there is no Dense Layer before the Activation part? $\endgroup$ – Predicted Life Oct 1 '18 at 0:06
  • $\begingroup$ So x = Activation('relu')(x) is only a transformation? $\endgroup$ – Predicted Life Oct 1 '18 at 0:10
  • $\begingroup$ Yes in this case you would simply be transforming the outputs of the previous layer. $\endgroup$ – JahKnows Oct 2 '18 at 2:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.