How to train the machine so that it can give 'out of bound/classes' as an output for neural network

I know I was not able to word the title of the question properly. So I am trying to explain the problem here: Suppose, I built and trained a CNN to identify numbers from 0 to 9. However, when I deployed the CNN, someone gave "#" as an input (anything but 0-9). What can I do to my neural network during training, so that it's output can say that this is none of the training characters?

I'd like to present a second example: instead of classifying, suppose we want to do denoising using an autoencoder. Again, the CNN autoencoder is trained for 0 to 9. Now, how can we prepare it so that if someone gives "\$"symbol (it can be anything that is not 0-9) as an input, it'll be able to identify that this symbol is none of what it was trained on? And the CNN autoencoder will be able to give an output according to that?

I think the best way would be to augment some data and have an additional output class "unknown". However, if that is not possible or the neural net can not be retrained I would compare the distribution of the outputs of a hidden layer.

For the CNN architecture below, calculate the empirical distribution for the outputs of a hidden layer after the flatten layer for the training data (e.g.$$n1$$). Save this distribution and during deployment, calculate the output of $$n1$$ for the test instance. If the output of $$n1$$ corresponds to a highly unlikely value with respect to the empirical distribution of training examples, return "unknown". Else, predict the digit for this instance. Three ideas come to my mind (from simple to complex)

1. Include an additional category for anything which is not a number and train your network on these $$k+1$$ categories.

2. Apply another predictor in the first place which has been trained to differentiate between "number" and "no number". Iff the input is classified as a number you then run your number recognition network. (this approach might make transfer learning easier, i.e. apply an existing model for the first step)

3. Merge the two tasks into one network and make this a multi-task classification, i.e. your network includes layers not only to recognize numbers but also for the binary classification "number vs no number". Since these tasks are closely related the two tasks might benefit from sharing parameters (i.e. use the same features). The paper An Overview of Multi-Task Learning in Deep Neural Networks describes this approach in more detail. (Note that this approach is not identical with the first idea in this list since this one applies two separate classifications while the first one only does a single classification)

However, as a disclaimer: I have not tried out the 3rd idea myself but used the second one for a similar problem.

• But the "unknown" digits are not in the training data, so none of your options work. Jan 7 '20 at 12:46