3
$\begingroup$

Bias in a neural network is an additional neuron to be fired i.e let $y=a+bx$ where $a$ is a bias term.

Do we have any difference between bias and inductive bias?

How Inductive bias is helpful in generalizing?

Any Kind of reference is helpful.

Thanks in Advance.

$\endgroup$
6
$\begingroup$

This is a tentative general answer.

Bias in statistics (and machine learning) is the difference between an estimated value and the true value. ie $\text{bias} = \hat{y}_{\text{estimated}} - y_{\text{true}}$.

Another (losely related) concept of bias especialy in mathematics is any, primarily, constant term that is added to something (and thus makes a difference between the two values and thus introduces bias). Bias in neural networks is meant in this mathematical sense, rather than the statistical one.

Apart from the above, bias as a term has been introduced in other fields, for example in psychology (eg cognitive bias). The term inductive bias comes from machine learning.

This sense of bias refers to the initial assumptions some entity or algorithm takes for granted and tries to learn based on them. So the induction made is influenced by these initial assumptions, and if these are proved wrong, then there will be bias in the usual statistical or mathematical sense, that is the estimation will differ from the true value.

In machine learning and statistics there is what is called Bias-Variance tradeoff meaning that the total Mean Squared Error (MSE) can be decomposed into bias (in the statistical sense), variance and irreducible error. How is this related to generalisation? Well to generalise well variance should be minimised (in most cases), so since irreducible error is out of control, one way is to increase bias, in the statistical sense (one way or another), if MSE is to be held the same.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.