Apologies for a naive question. Let's say I am training a simple feed-forward neural network using stochastic gradient descent with a fixed architecture, learning rate, number of training epochs, and batch size. I will randomly initialize my weights and biases by drawing them independently from a standard normal distribution.
Now let's say I run my training algorithm multiple times and obtain a sequence of weights and biases $((W_1, b_1), (W_2, b_2),..., (W_n, b_n))$ representing the trained models.
The different entries in this sequence will look very different (due to the random starting conditions as well as the stochasticity used in selecting the samples for training). One thing that ties them all together is that they all will all approximate a given function when used for calculations according to the architecture (assuming I have my hyperparameters set appropriately and the function I am attempting to model is being appropriately approximated by the trained models).
Are there other patterns that are sometimes observed in the $W_i$ (or $b_i$) as $i$ varies?
The sort of thing I could imagine would be, say, when plotting the Frobenius norms of the weight matrices divided by the norms of the respective bias vectors, as a function of the depth of the layer, the resulting graph might look similar across the different models. (I have not personally observed this, I just mention it to give a concrete example of the sort of thing I am looking for).
