I am starting with machine learning and I am currently trying to understand artificial neural networks. In a multi-layer perceptron one minimizes some cost function (it can e.g. cross-entropy), but I have to feed the neural network with all the training data I have. This is not what a typical human learning process looks like. Rather, the experience changes by providing new examples through time and the prediction may change due to novel informations. In other words, the machine/net has to response to the changing environment by refining its predictions. I have two questions:
Isn't it more natural to use Bayesian techniques for optimization where the next training example improves the experience (my neural net will improve through time by providing a new data)?
Does a neural net needs all the training data from the start or can I keep improving it by providing a new information (e.g. after some time) and without referring to the previous training data (I don't have to optimize everything once again but with larger collection of data)?