How to deal with variable number of permutation invariant features?

Question

I want to learn from data where each record has a variable number of features that have no inherent order to them. Take as an example the task to predict whether a repair is worth it of some item. Each item can have a variable amount of features e.g. glass is broken, handle is missing etc. which are mostly categorical. There are up to +1000 different things to know about each item, but in reality there are only a handful present.

What ways are there to encode this kind of data or what models exist that can handle this natively?

I have a few things in mind

• Binary encoding of set (resulting in a lot of dimensions)
• Average features into single vector (dimensions become harder to interpret)

but I want to explore more options.

Answer 1

Technically your problem is not about a variable number of features, since you can have a finite list of all the possible features. The standard case is just to use all these features, even if only a few of them are "active" for a particular instance (your first option). If the number of features is too high, then you need dimensionality reduction.

The advanced option is to use a feature extraction method: an unsupervised method which "groups" similar features together, the goal being to reduce dimensionality while preserving the information contained in the features. By definition this method modifies the set of features: the "new features" are not directly interpretable, as opposed to the original ones.

Note that the other way to reduce dimensionality is feature selection: these methods don't modify the features, but they discard the least informative ones. Feature selection is supervised (as opposed to feature extraction) because informativeness is measured with respect to the response variable.