1
$\begingroup$

I am a little confused about the term Bias and Weight with respect to machine learning.

Say we want to predict the heights of people whose weights are given. So plot weights to x-axis and height to yaxis. To find out the linear relationship between height and weight we draw a straight line that shows the relationship between height and weight.

enter image description here

Using the equation for a line, you could write down this relationship

$y= mx+b ...i)$ more specifically in the machine learning terminology it could be $y= b+ w_1x_1...ii)$

So here b is the bias as per machine learning. However, b is the y-intercept as per mathematics. By defination b can be defined in machine learning A value indicating how far apart the average of predictions is from the average of labels in the dataset. Is that mean bias(b) is the distance between some particular point of the red line(as per picture) to the true point(say a blue or green point).

Now another confusion if this is the case then what is loss? As per defination loss is A measure of how far a model's predictions are from its label. Then what is the difference between loss and bias?

Now, for weight, here, weight(m)means slope as per equation i). Mathematically slope can be define as

$m = \frac{rise}{run} =\frac{y_2 - y_1}{x_2 - x_1}$

However, weight$(w_1)$ can be defined in machine learning as A coefficient for a feature in a linear model. So my confusion is that is the procedure of finding weight is same as finding slope in mathematics?

$\endgroup$
1
$\begingroup$

Okay so let's start with the first question:

Is that mean bias(b) is the distance between some particular point of the red line(as per picture) to the true point(say a blue or green point).

You'd be correct had you used the word difference rather than distance. Bias is the difference between the estimated value and the true value. Think of it in this way, if your weighing machine always showed 5kg less for every measurement then you'd be adding 5 to every weight it measures, as a way to offset all measures made by the machine. So if your data is 'biased' towards one side then this bias term offsets this behaviour by adding/subtracting the 'biases of the weights. Here's some more rigorous explanation to it.

what is the difference between loss and bias?

Loss, on the other hand, is a measure of how close your estimator is to true prediction, there are different loss function that one can employ with different type of training, for linear regression we can use MSE. Using loss functions we can objectively optimize our weights and biases. So, having an appropriate value of bias(and weight) causes our loss to be lower, thereby increasing the accuracy of our model.

So my confusion is that is the procedure of finding weight is same as finding slope in mathematics?

The procedure is different, but the end result is the same. To see why you'd need to realize is that you are calculating $\hat y$ as the following: Given $m,b$ and a point $x$, calculate the value of $\hat y$ which is then used to calculate the loss. This value along with $m$ and $b$ is then used to calculate a better value for $m$ and $b$ using Gradient Descent. Notice at no point are we talking about directly calculating $m$ using $\hat y$, We are iteratively calculating better values for weights and biases. The end result? For single-dimension data, the weight that the model has learnt is indeed the slope (when plotted) and bias is the y-intercept. This stands true as the number of dimension increases but then rather than being single values, it becomes a multi-dimensional vector.

$\endgroup$
2
  • $\begingroup$ I am not sure about the difference between loss and bias. If bias means adding 5kg with every measurement to correct it, isn't it mean the difference between correct measurement and predicted measurement is 5Kg. For loss, if we measure how close our estimator with true prediction, isn't it a synonym of the difference between the estimator and true prediction? $\endgroup$
    – Encipher
    Mar 23 at 17:29
  • $\begingroup$ Suppose there isn't any weight, in that case, bias will be directly equal to $\hat y$. When that happens loss $L =\frac{1}{m}\sum_i(y_i-b)^2$. You'd see mathematically $b$ and $L$ are not the same. Bias is the same as weight if the $x$ associated with it was $1$. Bias is a learnable term that helps us reduce the loss, but vice-versa is not true. $\endgroup$ Mar 24 at 5:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.