What is weight and bias in deep learning?

Question

I'm starting to learn Machine learning from Tensorflow website. I have developed a very very rudimentary understanding of the flow a deep learning program follows (this method makes me learn fast instead of reading books and big articles).

There are a few confusing things that I have come across, 2 of them are:

1. Bias
2. Weight

In the MNIST tutorial on tensorflow website, they have mentioned that we need bias and weight to find the evidence of the existence of a particular pattern in an image. What I don't understand is, where and how the values for Bias and Weight are determined?

Do we have to provide these values or does the TensorFlow library calculates these values automatically based on the training data set?

Also if you could provide some suggestions on how to accelerate my pace in deep learning, that would be great!

Tensorflow Beginners Tutorial

Answer 1

Mathematically speaking, Imagine you are a model (No! not that kind, figure 8 ones).

Bias is simply how biased you are. If you are a Nigerian, and you are asked "Which nationality have the most beautiful women" you might say Nigerian Ladies. We can say it's because you are biased. So your formula is $Y = WX + nigerian$.

So what do you understand? Biased is that pre-assumption in a model like you have.

As for weight, logically speaking, Weight is your Gradient (as in linear algebra).

What is Gradient? it's the steepness of the Linear function.

What makes the linear gradient very steep (High positive value)?

It's because little changes in X(input) causes Large differences in Y axis(output). So you (Not as a Model anymore, but a brilliant Mathematician (your alter ego)) or your Computer tries to find this gradient, which you can call weight. The difference is that you use a pencil and graph book to find this, but the black box does its electronic Magic with registers.

In the Machine Learning Process, computers, or you, try to draw many straight lines or Linear functions across the data points.

Why do you try to draw many straight lines?

Because in your graph book/Computer memory, you are trying the see the line that fit appropriately.

How do I or Computer know the line that fits appropriately?

In my secondary school, I was taught to draw a line across the data points, visually checking the line that cuts through perfectly in the middle of all the data point.(Forget those A.I hype, our brains can calculate by just staring at things). But as for the computer, it tries the standard deviation and variance of each line towards the data points. The line with the least deviation (sometimes we call it the error function) is chosen.

Cool! so and what happens

The gradient of that line is calculated, let's say the Weight of the Learning problem is Calculated.

That's Machine Learning at its basic understand and a high school student plotting graph in his/her Graphbook.

Answer 2

I agree with the comments on your question that you should look into a course, maybe Andrew Ng's Machine Learning on Coursera, which is a highly regarded, free introductory course. This is a basic question about fundamentals of machine learning. As such I am not covering the maths in this answer, you can get that from many places, including that course.

where and how the values for Bias and Weight are determined?

Weights and biases are the learnable parameters of your model. As well as neural networks, they appear with the same names in related models such as linear regression. Most machine learning algorithms include some learnable parameters like this.

The values of these parameters before learning starts are initialised randomly (this stops them all converging to a single value). Then when presented with data during training, they are adjusted towards values that have correct output.

Do we have to provide these values or does the TensorFlow library calculates these values automatically based on the training data set?

You do not need to provide values before training, although you may want to decide things such as how many parameters there should be (in neural networks that is controlled by the size of each layer). TensorFlow calculates the values automatically, during training. When you have an already-trained model and want to re-use it, then you will want to set the values directly e.g. by loading them from file.

The specific code that handles changes to weights and biases from the tutorial is this:

train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)

and this:

sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})

The first line defines how the weights and values will be changed. You can read this almost literally as "define a training function that uses the gradient descent optimizer to reduce the cross entropy of the supplied data".

The second line invokes that function with a specific piece of data. Each time this second line is run, the weight and bias values are adjusted so that neural network outputs $y$ values a little bit closer to the correct association for each $x$ value.

Answer 3

Weight - Weight is the strength of the connection. If I increase the input then how much influence does it have on the output.

Weights near zero mean changing this input will not change the output. Many algorithms will automatically set those weights to zero in order to simplify the network.

Bias - as means how far off our predictions are from real values. Generally parametric algorithms have a high bias making them fast to learn and easier to understand but generally less flexible. In turn they are have lower predictive performance on complex problems that fail to meet the simplifying assumptions of the algorithms bias.

Low Bias: Suggests less assumptions about the form of the target function.

High-Bias: Suggests more assumptions about the form of the target function.