1
$\begingroup$

Forum contributor David Waterworth wrote: "we train $w_{0j}$", here $w_{0j}$ is the bias of an artificial neuron.

However, this Wikipedia article

https://en.wikipedia.org/wiki/Artificial_neural_network

(in section "Optimization", under subtitle "Algorithm")

seems to indicate that the bias does not adjust during training:

(I only quoted relevant sentences.)

"Let $N$ be a neural network with $e$ connections, $m$ inputs, and $n$ outputs.

$w_0, w_1, w_2,$ ... denote vectors in $R^e$. These are called weights.

The output of the algorithm is then $w_p$."

In other words, the training/optimization algorithm starts with $w_0$, then produces $w_1$, then produces $w_2$, etc.

Please note that $w_1, w_2, ...$, are $e$-dimensional vectors, and $e$ is the total number of connections.

So, does this Wikipedia article say that only the weights of the connections adjust, but the bias (weight of the neuron) does not adjust?

If the bias also adjusts, then the output vector of the training/optimization algorithm would have more dimensions - The number of dimensions would have to be:

[the total number of connections $e$] plus [the total number of neurons which have biases].

Perhaps the training/optimization algorithm has evolved since this Wikipedia article was written?

$\endgroup$
  • $\begingroup$ yes, bi also get adjusted during training. (bi = wi0 which is a weight) $\endgroup$ – asmgx Aug 8 at 5:45
1
$\begingroup$

Neurons values are something like this

$N_i = w_{i1}x_{i1} + w_{i2}x_{i2} + ... + w_{in}x_{in} + b_i$

the bias is $b$ here

but it also can be written like

$N_i = w_{i0} + w_{i1}x_{i1} + w_{i2}x_{i2} + ... + w_{in}x_{in}$

where bias is $w_{i0}$

it is the weight of index 0

when mentioned weights in optimization it is by default include bias

$\endgroup$
  • $\begingroup$ You have posted this as an edit "In the formula Ni=wi1xi1+wi2xi2+...+winxin+bi wi1,wi2,......,win would adjust during training. My question is: Does bi also adjust during training? " $\endgroup$ – asmgx Aug 8 at 5:44
  • $\begingroup$ yes, bi also get adjusted during training. (bi = wi0 which is a weight) $\endgroup$ – asmgx Aug 8 at 5:45
1
$\begingroup$

Now I can answer my own question:

Based on the comments of forum contributors David Waterworth and asmgx, the bias of an artificial neuron does get adjusted during training.

The Wikipedia article

https://en.wikipedia.org/wiki/Artificial_neural_network

was mistaken when it said the optimization produces a sequence of weights $w_0, w_1, ... , w_p$, where each $w_i$ is a vector in $R^e$, where e is the total number of connections in the artificial neuron network. Vectors of this size do not have place to hold the biases of the neurons.

The correct description should be:

Each $w_i$ is a vector in $ R^g $, where $g=e+f$, where $e$ is the total number of connections in the artificial neuron network, and $f$ in the total number of neurons that have biases.

This article

https://towardsdatascience.com/how-to-build-your-own-neural-network-from-scratch-in-python-68998a08e4f6

also agrees with David Waterworth and asmgx:

"The process of fine-tuning the weights and biases from the input data is known as training the Neural Network."

$\endgroup$
0
$\begingroup$

The $\omega$ in that section is a vector of weights not a single weight, and when they write $\omega_i$ in this context they mean the weights for every connection and all the bias's at iteration $i$ (i.e. the result of training a single step). It should really be written $\boldsymbol \omega_i$ or $\bar \omega_i$ to denote a vector but maybe Wikipedia doesn't support that?

So $\boldsymbol \omega = [\omega_{00}, \omega_{01}, \dots , \omega_{nk}]$

Also "and produces a sequence of weights $\omega_{0},\omega_{1},\dots ,\omega_{p}$ starting from some initial weight $\omega_{0}$" Could be cleared, i.e. "and produces a sequence of vectors at each iteration $\boldsymbol \omega_{1},\dots ,\boldsymbol \omega_{p}$ starting from some initial vector $\boldsymbol \omega_{0}$"

$\endgroup$
  • $\begingroup$ David, you wrote "when they write ωi in this context they mean the weights for every connection and all the bias's at iteration i". If you read the 2nd sentence in the "Algorithm" section in the Wikipedia article, you will see that w_1, w_2, ... , are vectors in R^e. R^e is the set of all vectors of e real numbers, where e is the total number of connections in the artificial neural network. So, each real number in w_1 is a single weight for a connection; therefore, there is no place left in that vector for the biases of neurons; therefore, each iteration doesn't seem to include biases changes. $\endgroup$ – Curious Aug 8 at 4:21
  • $\begingroup$ So they do, but I think either it's an oversight - i.e. here towardsdatascience.com/… both weights and bias's are optimised $\endgroup$ – David Waterworth Aug 8 at 6:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.