I was recently asked this question in an interview and wondered what the answer would be - "How do Factorization Machines get around the overfitting problem when using second-order interactions?"
$\begingroup$
$\endgroup$
asked Sep 19, 2017 at 2:48
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Here are some excerpts from the original paper that I think are key to understanding the question:
Instead of using an own model parameter for each interaction, the FM models the interaction by factorizing it. We will see later on, that this is the key point which allows high quality parameter estimates of higher-order interactions under sparsity.
Factorization machines can estimate interactions even in these settings (sparse data) well because they break the independence of the interaction parameters by factorizing them. In general this means that the data for one interaction helps also to estimate the parameters for related interactions.
In other words, instead of fitting an independent parameter for every second order interaction, it factorizes the parameters which reduces the parameter space and the model complexity, and thus making the model less prone to over fitting.
Hope this helps understand some aspects of the question.
$\begingroup$
$\endgroup$
Not mentioned in the original paper, you can add L2 regularization to the model. L2-reg can be added to 1st order weights and 2nd order weights as well. You can also limit the 2nd order rank when you don't have enough data.
