I think that it is important, in this situation, to ask yourself why you are using machine learning to detect the interactions. It feels a bit like data dredging. Using domain knowledge to think about what interactions would be likely or feasible given the phenomenon you are studying may well serve you well.
That said, you can fit a tree model using, for instance,
R. This will allow you to detect complex and high order interactions. You can see this as an example on page 30 here. Note that I would probably use
partykit for fitting the tree model because it allows you to use non-normal distributions. This method, though, works best if you have a very large data set and you may struggle to fit them in a linear model once you have found them (because they can only occur on one side of the tree).
An alternative method that addresses some of the limitations above is to use a technique called Additive Groves. These work on the principle of observing the relative performance of differently restricted tree models. Because interaction effects are not additive, this technique can allow for the identification of interactions.