I have a simple model, which learns well. It is a two tower recommender where we maximise dot product between positive pairs. The current structure is just an embedding layer followed by a dense layer with no nonlinearity.

To learn more complex patterns in the data I have tried to add an activation (relu) to the model, but this decreases model performance on both AUC and Factorized Top K.

So my question is, in what cases would adding an activation decrease the performance of the model? Does this mean my dataset is too simple to need a nonlinear activation?

I have read this issue - and I am trying both the Leaky RELU and Batch Norm now, but my problem is different as my model learns with the activation but achieves only around 70% of the performance without.


1 Answer 1


"The current structure is just an embedding layer followed by a dense layer with no nonlinearity." - According to this, your model is currently linear. If it falls short in its performance, you should start by adding another dense layer. Your overall structure will become:

  • embedding,
  • dense layer (followed by a ReLU activation)
  • A second dense layer.

Make sure the hidden layer is wide enough (i.e. has enough neurons) to pick up different patterns.

The problem you're describing with your current architecture is, most probably, that with ReLU as the final layer, you cannot output negative numbers (ReLU clips negative values to 0). For this reason, ReLUs are only very rarely used after the last layer of a deep learning model.

  • $\begingroup$ Hi, no sorry this is not the case currect structure is Emb() -> Dense() when trying Emb() -> Dense() -> RELU() -> Dense() performance decreases. $\endgroup$
    – dendog
    Commented Nov 23, 2022 at 10:32
  • $\begingroup$ Ahh, I see, in that case my current answer is not that useful since you've already implemented this. What are the shapes of each of the dense layers? Maybe you've made the hidden layer too small? $\endgroup$
    – KishKash
    Commented Nov 23, 2022 at 10:50
  • $\begingroup$ No that is not what the question is about - I am interested to abstractly/theoretically understand in what cases a nonlinearity could be detrimental. $\endgroup$
    – dendog
    Commented Nov 23, 2022 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.