I have a simple model, which learns well. It is a two tower recommender where we maximise dot product between positive pairs. The current structure is just an embedding layer followed by a dense layer with no nonlinearity.
To learn more complex patterns in the data I have tried to add an activation (
relu) to the model, but this decreases model performance on both AUC and Factorized Top K.
So my question is, in what cases would adding an activation decrease the performance of the model? Does this mean my dataset is too simple to need a nonlinear activation?
I have read this issue - and I am trying both the Leaky RELU and Batch Norm now, but my problem is different as my model learns with the activation but achieves only around 70% of the performance without.