What is the best way to train a neural network with a variable number of inputs?

Question

Suppose I have a neural network with 5 inputs: [A,B,C,D,E]

There is only 1 output. The expected accuracy of the model should increase when all 5 inputs are available, but often not all 5 inputs are available. For example, I might have cases where I only have a variable number of the inputs, e.g. [A,B,C,-,-], [A,-,-,-,E], [-,B,-,D,-], [A,-,-,-,-], [-,-,C,-,-], [A,-,C,D,E], etc.

In such a situation, what is the best way to train or build the neural network? Are there any specific approaches or architectures recommended for this type of problem?

A couple ideas that come to mind include:

1. Double the number of inputs to the neural network by including a second "binary input vector" that determines whether the input variable is present or not. For example, the binary input vector for the inputs [A,-,C,-,E] would simply correspond to [1,0,1,0,1], which could be fed into the neural network as well. The outstanding question is how does one treat the undefined variables with "-" as placeholders in such an example...perhaps defaulting to 0 for "-" is one naive but simple way when coupling the binary vector.

2. Build and train separate neural network for every combination of [A,B,C,D,E] — this could certainly be implemented, but would be a brute force approach that requires a lot of training and be rather inefficient. For 5 variables, this would require 31 separate neural networks, and would scale poorly as the number of potential input variables increase

Answer 1

The expected accuracy of the model should increase when all 5 inputs are available

Not necessarily true, NNs do the learning, and may decide that the most import features are A,C,D, and the others only contribute a small amount, while I'm not saying they aren't important, it isn't necessary that missing B and/or E even alters the output.

With option 2, you've already addressed a major problem with that approach. I work with many networks that have input dimensions >1000, so this clearly cannot scale well. Don't think I need to add much more you've solved that already.

For option 1, that is a rather intuitive idea, why not just teach the network what is missing by showing it a 0? Well the problem there is that 0 is a meaningful value, and is significantly different from a missing value. So the network would still be learning a great deal from that input, which sometimes is good, but if it is also possible for say Feature B to be 0, then that is overlapping the value with a biased (not chosen by random) replacement for it.

One of the fantastic things about Neural Networks is their ability to learn from data, and this includes when data is missing from the input. Now, it is important to recognise that in training you should be remaining as balanced as you can, if Feature B only exists in say 5% of the training data, is it worth including in the feature space of the model if it can't add too much information? There are techniques to balance this data appropriately, and there are a lot of methods to impute, replace, or predict missing values.

A fairly significant amount of ML techniques can handle missing values, so often you won't have to worry about them. For example, a RandomForest Classifier will be able to ignore/handle missing values, while still learning effectively.

When it comes down to it, you should pick a feature space that is reasonably stable, if only 5% of all training data contains all 5 inputs, then 5 features isn't likeley to be the best choice, find a happy medium where you do have a significant number of features present in all training data.

Answer 2

I would just set your missing values in the input layer to zero, or equivalently, apply your binary vector as a mask to the input layer. This would be in effect the same as applying dropout to your input layer, but with a crucial difference - instead of data being dropped randomly the actual nature of the data determines which data are dropped. This may actually be important and therefore you are giving the network an opportunity to learn this.

I disagree with fam-woodpecker when they say "that 0 is a meaningful value" in the sense that introducing zero implies introducing unwarranted, spurious meaning and therefore should be avoided. As explained above, it would be like using dropout, and there is a large body of positive evidence about this in terms of regularisation, sparsity, ensembles of networks etc.

Response to comment: The diagram at the top of the second page of the original paper clearly shows that dropout is applied to the input layer as well as hidden layer(s). I have no idea about the effect of applying dropout to the input layer only. I would suggest applying dropout to the hidden layer(s) as that is what I meant about the above mentioned positive evidence, with the twist that the dropout to the input layer has already been applied by the virtue of the missing input data being set to a zero value. During training, any flag to apply dropout to the input layer should be set to false as your data preprocessing (setting zeros) would have already taken care of this.