1
$\begingroup$

While training the neural network (or any other supervised learning algorithms), we supply input variables and corresponding outputs. The input variables can be continuous or discrete (binary in many cases).

What happens if after training with a binary input data, we supply a continuous value for the same input at the time of evaluation? Does the algorithm internally treat all variables as continuous variables?

For example, suppose one of the inputs is Young/Old encoded in the form of 0/1 in the training dataset. What happens if we supply a value of, say, 0.2 at the prediction stage? Does it/should it make any sense to the network?

$\endgroup$
1
$\begingroup$

I guess it depends on the algorithm, but linear models, as well as neural networks, treat all variables as continuous. The algorithm will not explode or anything if you supply 0.2 at prediction stage. However, your algorithm is trained on data. The algorithm can at best do what it has learnt from the training data. For this reason, do not expect anything meaningful when you feed an example with a value that has not been seen in the whole training set, or that it does not follow the training set distribution.

| improve this answer | |
$\endgroup$
  • $\begingroup$ "do not expect anything meaningful when you feed an example with a value that has not been seen in the whole training set". I do not think this is true. If I have a variable AGE for which the training set has different valid values in the range [1,100], any new value within the same range at the time of prediction will provide a meaningful result. Of course a value like AGE=300 may not be meaningful. $\endgroup$ – Saptarshi Roy May 8 '18 at 8:07
  • $\begingroup$ Yeah, but there is an order in here. With categorical variables there isn't any particular order. Take a tree-based model: you know that it has done a split based on a binary variable, but you don't really know if the split has been done with 0.5 as a threshold, so you don't really know how will the split act on 0.2. $\endgroup$ – David Masip May 8 '18 at 8:20
  • $\begingroup$ But you are right, what I meant is that the algorithm learns from a particular distribution, and if your test set does not have the same distribution then you are very likely to overfit. $\endgroup$ – David Masip May 8 '18 at 8:22
  • $\begingroup$ If the algorithm treats all variables as continuous variables, then all variables including categorical ones should have an order (after converging to the dummy variables). This is exactly the point I am confused about. Even if only 0 and 1 are fed as input, the range in between also should be a valid input. Alternatively, we know that any age value between [0,100] is valid. But if we had used only (10, 20, 30, 40, 50, 60, 70 ,80, 90) age values in the training set, would the algorithm consider 23, 56 or 89 as valued out of the distribution? $\endgroup$ – Saptarshi Roy May 8 '18 at 8:55
  • $\begingroup$ The range in between will be a valid input. However, when models are nonlinear you don't really know what to expect about the predictors are in that range. $\endgroup$ – David Masip May 8 '18 at 9:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.