In what cases does the addition of a nonlinearity decrease neural network performance?

Question

I have a simple model, which learns well. It is a two tower recommender where we maximise dot product between positive pairs. The current structure is just an embedding layer followed by a dense layer with no nonlinearity.

To learn more complex patterns in the data I have tried to add an activation (relu) to the model, but this decreases model performance on both AUC and Factorized Top K.

So my question is, in what cases would adding an activation decrease the performance of the model? Does this mean my dataset is too simple to need a nonlinear activation?

I have read this issue - and I am trying both the Leaky RELU and Batch Norm now, but my problem is different as my model learns with the activation but achieves only around 70% of the performance without.

Answer 1

"The current structure is just an embedding layer followed by a dense layer with no nonlinearity." - According to this, your model is currently linear. If it falls short in its performance, you should start by adding another dense layer. Your overall structure will become:

• embedding,
• dense layer (followed by a ReLU activation)
• A second dense layer.

Make sure the hidden layer is wide enough (i.e. has enough neurons) to pick up different patterns.

The problem you're describing with your current architecture is, most probably, that with ReLU as the final layer, you cannot output negative numbers (ReLU clips negative values to 0). For this reason, ReLUs are only very rarely used after the last layer of a deep learning model.