0
$\begingroup$

Consider a simple Matrix Factorization or any embedding algorithm. It is obvious that a very small embedding dimension does not contain enough information so the algorithm will have low performance and enlarging the dimension will improve the performance. However, as I tried various algorithms, enlarging the dimension more than a threshold decreases the performance of the algorithm. Why does this happen?

I know using larger embedding while you can get the same result with smaller ones does not make sense, however, I am just curious to find out more about that.

$\endgroup$
1
$\begingroup$

You are probably just overfitting.

When you increase the number of parameters that are being trained in your model, the likelihood of them learning some aspect of the training data that doesn't generalize to unseen data increases. So there should be some sweet spot, where the model is able to express the underlying structure in the data sufficiently, without putting too much emphasis on those aspects that are specific to the training data.

$\endgroup$
1
  • $\begingroup$ You are right, large models are vulnerable to overfitting, however in my experiments, the training loss of a very large model is not better than training loss of smaller models. I expected that training loss gets better with enlarging the model. $\endgroup$
    – Taher
    Apr 3 '20 at 6:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.