# Nested features with one to many relationships

I have a dataset that has (among others) a categorical variable with many levels and further attributes associated with each level.

For example, consider predicting machine failure based on its last repair report.

• A machine has many (many) different parts.
• Every part could have been damaged or not.
• If damaged, it could have been inspected, repaired or replaced etc.
• These operations have associated costs and the part itself has a money value and other attributes (like the time it took to repair it...).

If I want to make predictions on the level of a machine, I need to aggregate all this information. I could one-hot encode all the parts indicating whether they have been damaged or not. But this still leaves me with the other attributes on the lower part level, like price and performed operation. I could probably further expand them into individual columns by considering all the combinations like part_X-repaired-..-price, part_X-replaced-..-price, ... part_Z-replaced-..-price but this seems to get out of hand.

Is there a better way to handle this type of data? I was thinking maybe some clustering technique but when I try to set it up I run into the same problem.

It is somewhat a reversed hierarchical model structure (in hierarchical (linear) models the outcome variable is on the lowest level if I'm not mistaking).

• The most obvious formalism is a graphical model, but they're a bit involved and scalability can be an issue. There's a great course on Coursera if you're not familiar. For implementation you should look into a new library called edward. If that sounds like too much work, just expand the tree like you suggested. – Emre May 25 '17 at 22:55

This is called "bilevel optimization" where one problem is embedded (nested) within another. If there are more than one objectives at one or both levels, it is then called "multi-objective bilevel optimization". There is a rich literature on this problem, including the book Multilevel Optimization: Algorithms and Applications.

Finding solutions to minimize the objectives often does not require machine learning, just linear programming.

• Thanks for your answer. If the "outer" problem is predicting machine failure, what would the embedded problem be? Not quite sure that's the right tool here... – oW_ Nov 13 '17 at 16:44
• The embedded problem would be predicting part failure. Some parts might be shared across machines then you could attribute aspects of machine parts to specific part failure patterns. – Brian Spiering Nov 18 '17 at 17:01

I could probably further expand them into individual columns by considering all the combinations like part_X-repaired-..-price, part_X-replaced-..-price, ... part_Z-replaced-..-price but this seems to get out of hand.

That is the way I would go if the approach you are using doesn't like high cardinality categorical variables. There might be some interesting relationships that fall out, like part A tends to need a repair when part B is replaced with a cheap version. If you end up with too many variables, you could try something like PCA to reduce them.

 Thought of another thing to look into: Vowpal Wabbit. It is an algorithm targeted almost exactly at your problem - lots of sparse variables. They claim it can handle 1 billion.

• From my understanding, the problem involves machine parts which are discrete. PCA is not the appropriate tool for discrete-valued dimension reduction. Multiple correspondence analysis is more useful for discrete-valued dimension reduction. – Brian Spiering Dec 13 '17 at 17:47