Keras functional api explanation of activation() layer?

Question

With the keras functional api it is possible to write something like this:

x = Activation('relu')(x)
x = Dense(8, activation='softmax')(x)

My question is whether the Activation() function is a separate Layer (equivalent to Dense(128, activation='relu') or if not... why and when this notation is used?

Answer 1

As stated in the docs, the activation layer in keras is equivalent to a dense layer with the same activation passed as an argument.

This would be equivalent

x = Dense(64)(x)
x = Activation('relu')(x)

is equivalent to

x = Dense(8, activation='relu')(x)

---

As per your example if the activation layer is used as a layer, this will act as a transformation of the outputs of the previous layer.

As you can see in the model

inputs = Input(shape=(784,))
x = Dense(32, activation='tanh')(inputs)
x = Activation('relu')(x)
predictions = Dense(8, activation='softmax')(x)

The output of the first layer is the result of a densely connected layer with a tanh function. Then these outputs will each be transformed by a relu function. You can see how this is simply a transformation and does not introduce any new model parameters using

model.summary()