Perhaps we can forget the graph idea to start and just go through the mathematics of it and then we will map our computations to a type of graph just so that it makes it easier to scale.
One neuron network is called the perceptron
The first neural networks only had a single neuron which took in some inputs $x$ and then provide an output. A common function used is the sigmoid function
$\sigma(z) = \frac{1}{1+exp(z)}$
$\sigma(w^Tx) = \frac{1}{1+exp(w^Tx + b)}$
where $w$ is the associated weight for each input $x$ and we have a bias $b$.
How do we train these weights
We will use gradient descent to train the weights based on the output of the sigmoid function and we will use some cost function $C$ and train on batches of data of size $N$.
$C = \frac{1}{2N} \sum_i^N (\hat{y} - y)^2$
$\hat{y}$ is the predicted class obtained from the sigmoid function and $y$ is the ground truth label. We will use gradient descent to minimize the cost function with respect to the weights $w$. To make life easier we will split the derivative as follows
$\frac{\partial C}{\partial w} = \frac{\partial C}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial w}$.
$\frac{\partial C}{\partial \hat{y}} = \hat{y} - y$
and we have that $\hat{y} = \sigma(w^Tx)$ and the derivative of the sigmoid function is $\frac{\partial \sigma(z)}{\partial z} = \sigma(z)(1-\sigma(z))$ thus we have,
$\frac{\partial \hat{y}}{\partial w} = \frac{1}{1+exp(w^Tx + b)} (1 - \frac{1}{1+exp(w^Tx + b)})$.
So we can then update the weights through gradient descent as
$w^{new} = w^{old} - \eta \frac{\partial C}{\partial w}$
where $\eta$ is the learning rate.
What if we have 2 layers?
Take a look at the following answer: Compute backpropagation
Why model neural networks as a graph
In graph theory nodes are connected by vertices. Each neuron takes in some inputs and produces an output. The inputs to this node are not the same as the outputs of the previous layer. They are affected by some weight. Thus we can consider these weights to be the vertices of the graph that link nodes together.
