Does the bias of an artificial neuron adjust or remain constant during training?

Question

Forum contributor David Waterworth wrote: "we train $w_{0j}$", here $w_{0j}$ is the bias of an artificial neuron.

However, this Wikipedia article

https://en.wikipedia.org/wiki/Artificial_neural_network

(in section "Optimization", under subtitle "Algorithm")

seems to indicate that the bias does not adjust during training:

(I only quoted relevant sentences.)

"Let $N$ be a neural network with $e$ connections, $m$ inputs, and $n$ outputs.

$w_0, w_1, w_2,$ ... denote vectors in $R^e$. These are called weights.

The output of the algorithm is then $w_p$."

In other words, the training/optimization algorithm starts with $w_0$, then produces $w_1$, then produces $w_2$, etc.

Please note that $w_1, w_2, ...$, are $e$-dimensional vectors, and $e$ is the total number of connections.

So, does this Wikipedia article say that only the weights of the connections adjust, but the bias (weight of the neuron) does not adjust?

If the bias also adjusts, then the output vector of the training/optimization algorithm would have more dimensions - The number of dimensions would have to be:

[the total number of connections $e$] plus [the total number of neurons which have biases].

Perhaps the training/optimization algorithm has evolved since this Wikipedia article was written?

Answer 1

Neurons values are something like this

$N_i = w_{i1}x_{i1} + w_{i2}x_{i2} + ... + w_{in}x_{in} + b_i$

the bias is $b$ here

but it also can be written like

$N_i = w_{i0} + w_{i1}x_{i1} + w_{i2}x_{i2} + ... + w_{in}x_{in}$

where bias is $w_{i0}$

it is the weight of index 0

when mentioned weights in optimization it is by default include bias

Answer 2

Now I can answer my own question:

Based on the comments of forum contributors David Waterworth and asmgx, the bias of an artificial neuron does get adjusted during training.

The Wikipedia article

https://en.wikipedia.org/wiki/Artificial_neural_network

was mistaken when it said the optimization produces a sequence of weights $w_0, w_1, ... , w_p$, where each $w_i$ is a vector in $R^e$, where e is the total number of connections in the artificial neuron network. Vectors of this size do not have place to hold the biases of the neurons.

The correct description should be:

Each $w_i$ is a vector in $ R^g $, where $g=e+f$, where $e$ is the total number of connections in the artificial neuron network, and $f$ in the total number of neurons that have biases.

This article

https://towardsdatascience.com/how-to-build-your-own-neural-network-from-scratch-in-python-68998a08e4f6

also agrees with David Waterworth and asmgx:

"The process of fine-tuning the weights and biases from the input data is known as training the Neural Network."

Answer 3

The $\omega$ in that section is a vector of weights not a single weight, and when they write $\omega_i$ in this context they mean the weights for every connection and all the bias's at iteration $i$ (i.e. the result of training a single step). It should really be written $\boldsymbol \omega_i$ or $\bar \omega_i$ to denote a vector but maybe Wikipedia doesn't support that?

So $\boldsymbol \omega = [\omega_{00}, \omega_{01}, \dots , \omega_{nk}]$

Also "and produces a sequence of weights $\omega_{0},\omega_{1},\dots ,\omega_{p}$ starting from some initial weight $\omega_{0}$" Could be cleared, i.e. "and produces a sequence of vectors at each iteration $\boldsymbol \omega_{1},\dots ,\boldsymbol \omega_{p}$ starting from some initial vector $\boldsymbol \omega_{0}$"