# 3-way factorization machines

Factorization machines are a great way to model interactions between different features, in a way that is more efficient than full-blown quadratic interactions. There exist efficient implementations like libFM and Vowpal Wabbit's --lrq option.

My question is about higher order interactions: the FM paper mentions that it is "easy" to extend it to higher orders (Tensor FM?), see. Eq. (5) in the paper. Are there any good implementations of 3-way factorization machines?

[moved from cross-validated]

• You should not cross-post. As you have already asked in CV, either you can delete it there and let it remain here, else delete this. – Dawny33 Sep 24 '15 at 17:33
• @Dawny33 got it. deleted the CV question. – Amir Sep 24 '15 at 17:48
• If no one jumps on with a 3rd order or Nth order implementation, I would suggest that you just feed the 2nd order factors and 1st order back into Vowpal Wabbit's --lrq which will create all of the 3rd order pairings. You can repeat this ad infinitum to get an Nth order FM. – AN6U5 Sep 24 '15 at 18:43
• @AN6U5 I'm not sure I am following. Are you proposing to do this in several learning steps? How else would you obtain these "2nd order" factors? the second order factors are the result of the learning algorithm.. can you give an example? Suppose you have three namespaces A,B,C, each with 1000 different feature values. 3-way FM would learn a length k vector for each of the features (3*1000*k parameters). How would you get this otherwise? – Amir Sep 24 '15 at 18:48

I have recently started polylearn, a scikit-learn-contrib package for efficient implementation of factorization machines and polynomial networks, in Python with Cython. Polylearn currently coordinate descent solvers for 2nd and 3rd order factorization machines, and for arbitrary order polynomial networks, using the approach described in the papers cited on the polylearn website.

I'm currently working on an implementation of arbitrary higher order FMs using SG and CD style algorithms, using the approach from this recent paper, I should have it ready within a week.

Polylearn has not had a release yet, so I would love to hear any feedback on the API and functionality, in order to round things off before the 0.1 release!

A Tensor Flow implementation https://github.com/geffy/tffm/

It supports arbitrary order (>=2) Factorization Machines, dense and sparse input. The interface is in Scikit-learn style.

• This is essentially a link-only answer, which is discouraged. Elaborate with some detail of how this addresses the problem. – Sean Owen May 3 '16 at 20:02
• As far as I understood, Amir asked for an implementation of 3-order FMs. For some reason, it was not implemented anywhere, that's why we implemented it and I provided the link in case he still needs it. What would make the answer better? I can for example tell about the implementation details, but I'm not sure if that is of interest. – Bihaqo May 4 '16 at 9:43
• What is missing in the answer is whether the implementation is "good" - it is currently unclear (i.e. no examples, no unit tests etc.). It would be interesting to get some more details / experimental results. – Amir May 4 '16 at 18:14
• OK, sparse data support and examples/experiments are on the way, stay tuned! – Bihaqo May 5 '16 at 11:38