4
$\begingroup$

I was recently asked this question in an interview and wondered what the answer would be - "How do Factorization Machines get around the overfitting problem when using second-order interactions?"

$\endgroup$
6
$\begingroup$

Here are some excerpts from the original paper that I think are key to understanding the question:

  • Instead of using an own model parameter for each interaction, the FM models the interaction by factorizing it. We will see later on, that this is the key point which allows high quality parameter estimates of higher-order interactions under sparsity.

  • Factorization machines can estimate interactions even in these settings (sparse data) well because they break the independence of the interaction parameters by factorizing them. In general this means that the data for one interaction helps also to estimate the parameters for related interactions.

In other words, instead of fitting an independent parameter for every second order interaction, it factorizes the parameters which reduces the parameter space and the model complexity, and thus making the model less prone to over fitting.

Hope this helps understand some aspects of the question.

|improve this answer|||||
$\endgroup$
0
$\begingroup$

Not mentioned in the original paper, you can add L2 regularization to the model. L2-reg can be added to 1st order weights and 2nd order weights as well. You can also limit the 2nd order rank when you don't have enough data.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.