0
$\begingroup$

I have a problem. I have a CNN model which is used for an NLP problem. This is written in Python. I have questions about this, which I can't find an answer to.

  • Why is ReLu used inside the Conv1D layer and not Softmax ?
  • Why is ReLu used again as activation function in the first Dense-Layer and why Softmax afterwards ?
model1 = Sequential()

model1.add(
        Embedding(vocab_size
                ,embed_size
                ,weights = [embedding_matrix] #Supplied embedding matrix created from glove
                ,input_length = maxlen
                ,trainable=False)
         )
model1.add(Conv1D(256, 7, activation="relu"))
model1.add(MaxPooling1D())
model1.add(Conv1D(128, 5, activation="relu"))
model1.add(MaxPooling1D())
model1.add(GlobalMaxPooling1D())
model1.add(Dense(128, activation="relu"))
model1.add(Dense(number, activation='softmax'))
print(model1.summary())
```
$\endgroup$
0

1 Answer 1

2
$\begingroup$

The softmax activation is used as the activation function of the last layer in multiclass classification problems because it gives a categorical probability distribution over N discrete options.

ReLU is used as a middle-layer (either convolution or dense) activation function because it is a non-linearity that works well and is robust to the vanishing gradient problem (as opposed to tanh or sigmoid).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.