4
$\begingroup$

Suppose I have a neural network with 5 inputs: [A,B,C,D,E]

There is only 1 output. The expected accuracy of the model should increase when all 5 inputs are available, but often not all 5 inputs are available. For example, I might have cases where I only have a variable number of the inputs, e.g. [A,B,C,-,-], [A,-,-,-,E], [-,B,-,D,-], [A,-,-,-,-], [-,-,C,-,-], [A,-,C,D,E], etc.

In such a situation, what is the best way to train or build the neural network? Are there any specific approaches or architectures recommended for this type of problem?

A couple ideas that come to mind include:

  1. Double the number of inputs to the neural network by including a second "binary input vector" that determines whether the input variable is present or not. For example, the binary input vector for the inputs [A,-,C,-,E] would simply correspond to [1,0,1,0,1], which could be fed into the neural network as well. The outstanding question is how does one treat the undefined variables with "-" as placeholders in such an example...perhaps defaulting to 0 for "-" is one naive but simple way when coupling the binary vector.

  2. Build and train separate neural network for every combination of [A,B,C,D,E] — this could certainly be implemented, but would be a brute force approach that requires a lot of training and be rather inefficient. For 5 variables, this would require 31 separate neural networks, and would scale poorly as the number of potential input variables increase

$\endgroup$
1
  • 1
    $\begingroup$ Yes, idea 1 is generally a much better than idea 2, and usually better than trying to find some magic number to represent NA values. $\endgroup$ Mar 25 at 17:33

2 Answers 2

3
$\begingroup$

The expected accuracy of the model should increase when all 5 inputs are available

Not necessarily true, NNs do the learning, and may decide that the most import features are A,C,D, and the others only contribute a small amount, while I'm not saying they aren't important, it isn't necessary that missing B and/or E even alters the output.

With option 2, you've already addressed a major problem with that approach. I work with many networks that have input dimensions >1000, so this clearly cannot scale well. Don't think I need to add much more you've solved that already.

For option 1, that is a rather intuitive idea, why not just teach the network what is missing by showing it a 0? Well the problem there is that 0 is a meaningful value, and is significantly different from a missing value. So the network would still be learning a great deal from that input, which sometimes is good, but if it is also possible for say Feature B to be 0, then that is overlapping the value with a biased (not chosen by random) replacement for it.

One of the fantastic things about Neural Networks is their ability to learn from data, and this includes when data is missing from the input. Now, it is important to recognise that in training you should be remaining as balanced as you can, if Feature B only exists in say 5% of the training data, is it worth including in the feature space of the model if it can't add too much information? There are techniques to balance this data appropriately, and there are a lot of methods to impute, replace, or predict missing values.

A fairly significant amount of ML techniques can handle missing values, so often you won't have to worry about them. For example, a RandomForest Classifier will be able to ignore/handle missing values, while still learning effectively.

When it comes down to it, you should pick a feature space that is reasonably stable, if only 5% of all training data contains all 5 inputs, then 5 features isn't likeley to be the best choice, find a happy medium where you do have a significant number of features present in all training data.

$\endgroup$
3
  • $\begingroup$ Thank you for these thoughts! Regarding option #1, whilst I do agree that "0 is a meaningful value," the idea is that this second vector would be strictly composed of binary 1s and 0s (and perhaps when the second vector is zero, which indicates that the input is missing, the initial vector's input could be set to -inf or some large value out of the distribution) -- would this approach not work? $\endgroup$
    – user18959
    Mar 19 at 12:51
  • $\begingroup$ Learning from a missing value, and learning from, as you suggested, -inf would likely not end up being ultimately different. However, -inf can cause some numerical instability since a NN is just a long sequence of calculations, hence you would rather ignore an input than provide a value to it (in this case). Additionally, if it is something such as -inf, the model would likely teach itself to ignore that entirely and then there is not much importance to the feature anyway, but maybe that feature being present is actually going to be important. $\endgroup$ Mar 20 at 2:17
  • $\begingroup$ I think another crucial thing with respect to the data rather than the model itself is to consider why the feature is missing. If it is missing by design then it may not be a good idea to include the feature. However if the feature is missing because of data quality issues, then the other answers have suggested imputation. These two situations have slightly different meanings. $\endgroup$
    – timmy1691
    Mar 25 at 14:05
1
$\begingroup$

I would just set your missing values in the input layer to zero, or equivalently, apply your binary vector as a mask to the input layer. This would be in effect the same as applying dropout to your input layer, but with a crucial difference - instead of data being dropped randomly the actual nature of the data determines which data are dropped. This may actually be important and therefore you are giving the network an opportunity to learn this.

I disagree with fam-woodpecker when they say "that 0 is a meaningful value" in the sense that introducing zero implies introducing unwarranted, spurious meaning and therefore should be avoided. As explained above, it would be like using dropout, and there is a large body of positive evidence about this in terms of regularisation, sparsity, ensembles of networks etc.

Response to comment: The diagram at the top of the second page of the original paper clearly shows that dropout is applied to the input layer as well as hidden layer(s). I have no idea about the effect of applying dropout to the input layer only. I would suggest applying dropout to the hidden layer(s) as that is what I meant about the above mentioned positive evidence, with the twist that the dropout to the input layer has already been applied by the virtue of the missing input data being set to a zero value. During training, any flag to apply dropout to the input layer should be set to false as your data preprocessing (setting zeros) would have already taken care of this.

$\endgroup$
1
  • $\begingroup$ The large body of positive evidence regarding dropout is based on using it for the hidden layers of a neural network, correct? Is there any evidence to suggest it would work directly in the input layer? $\endgroup$
    – user18959
    Mar 25 at 7:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.