1
$\begingroup$

Suppose that we are training a linear regressor (perceptron). Adding extra features that are not related to the target (e.g. randomly generated values) before training will typically ____ our training error.

Would it typically increase, decrease or stay the same?

Listed Answer: decrease

The answer to this question is listed as decrease, but wouldn't adding extra unrelated features not change the training error, because the model would just learn to ignore those features (add 0 weights to them). Even if the model didn't learn to ignore the extra features, if anything wouldn't the training error increase as it doesn't seem like the model would be able to learn a relationship that would benefit the training error from a completely random unrelated feature?

$\endgroup$
1
$\begingroup$

If you add a new feature to the perceptron, the perceptron actually gets one more parameter. So in some sense it is not the same model anymore. But lets ignore that.

Correlation is the key problem. Assume the new feature is a random number which happens to have positive correlation with the desired output for the training dataset, but negative correlation with the test dataset. Then you would expect this new feature to influence the test error negatively, right? At least I would.

But your question was about the training error. In that case I actually don't see a reason why it should be the case. I'd suggest to try it.

And anyways, ask the person/organisation, where you found this question. (And post the answer back)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.