How to handle features which are not always available?

I have a feature in my feature vector that is not always available respectively sometimes (for some samples) it makes no sense to use it. I feed a sklearn MLPClassifier with this feature vector. Does the neural network learn by itself when the feature make sense to use for its decision or do I have to add a flag in the feature vector which says for example “1” if it makes sense or “0” if it does not make sense.

If the feature doesn't make sense in a subset of the samples, doesn't this mean that this is (or should be) a separate dataset, that needs a second model? That's one approach I'd think about.

The second would be to work with the data (feature) itself. It's probably best to use neutral value.

• In case of numerical value:
• try using a mean or median value, calculated on all entries
• try using an extreme value, e.g. -1 if your feature has only positive values. This should indicate that the feature is missing and the network should be able to handle it.
• In case of textual value, e.g. word embeddings, replace the value with a placeholder like N/A that doesn't have an embedding

If the features you train with are not the same you want to predict with you have a couple of options:

1. Retrain the model such that the feature in question is not used, since it won't be in your prediction data set.
2. Impute some value for that feature if it is missing in your data set. In your example you might make an assumption that it is "0" if it is missing, but you will have to decide this based on the data set and your intuition.

If the feature vector lengths of your training set and prediction set are different, then you are going to run into errors on the prediction set.

Depending on the problem, you can solve the problem by deleting these values and imputing them with an estimate if it's possible. Another strategy is to scale features on a -1,1 scale and impite this values with for example -3. Then use some robust methods that completely ignore the "outliers". Hope this helps.

You can use dummy variable encoding if the cases. You can enhance this idea to your problem as well. I will illustrate the procedure for a simple linear regression.

Imagine we want to predict the income of a person $$y_i$$ by using years of education $$x_{1i}$$, lectures taught $$x_{2i}$$, papers published $$x_{3i}$$ and current academic position $$x_{4i}$$. The sample does contain academic as well as non-academic persons.

1. Alternative: Assign natural void values. E.g. If we are looking for a child it does not make sense to include the income. But Income has a natural void value which is $$0$$. You could check if your variables also allow such a void value.

2. Alternative: You could split the dataset into two groups (academic and non-academic). And run two separate models.

3. Alternative: Introduces a new dummy variable is_academic $$x_{5i}$$ this variable is $$0$$ if the person $$i$$ is not academic and the value is $$1$$ if the person $$i$$ is academic. Then your regression model would look like

$$y_i = w_0+\tilde{w}_0x_{5i}+w_1x_{1i}+\tilde{w}_1x_{5i}x_{1i}+\tilde{w}_2x_{5i}x_{2i}+\tilde{w}_3x_{5i}x_{3i}++\tilde{w}_4x_{5i}x_{4i}+\varepsilon_i$$

So our data set is not $$x_{1i}, x_{2i}, x_{3i}, x_{4i}, y_i$$ but $$x_{1i},x_{5i},x_{5i}x_{1i},x_{5i}x_{2i},x_{5i}x_{3i}, x_{5i}x_{4i}, y_i$$.Now the dataset is complete but the model is not using linear basis function anymore.

Similarly, you could think about your dataset and introduce dummy variables when you see that some features are only present/useful for one subsample in your dataset.