I am starting with machine learning and I am currently trying to understand artificial neural networks. In a multi-layer perceptron one minimizes some cost function (it can e.g. cross-entropy), but I have to feed the neural network with all the training data I have. This is not what a typical human learning process looks like. Rather, the experience changes by providing new examples through time and the prediction may change due to novel informations. In other words, the machine/net has to response to the changing environment by refining its predictions. I have two questions:

  • Isn't it more natural to use Bayesian techniques for optimization where the next training example improves the experience (my neural net will improve through time by providing a new data)?

  • Does a neural net needs all the training data from the start or can I keep improving it by providing a new information (e.g. after some time) and without referring to the previous training data (I don't have to optimize everything once again but with larger collection of data)?

  • $\begingroup$ For neural networks you need to train the old information alongside with the new information, otherwise your network will converge to only the new information and not the old information. But this depends on how much you train your network the new information (e.g. if your information is 10% of the total size of the information, and you only train it 100 epochs with learning rate 0.01 it isn't always necessary to train the old information). $\endgroup$ – Thomas W May 4 '17 at 11:02
  • $\begingroup$ All machine learning models improve with more data, not just Bayesian ones. They can also be trained "online", as the data arrives. There are debates about biologically accurate artificial neural networks, but your question is more generic. $\endgroup$ – Emre May 4 '17 at 15:54

In my opinion the learning of MLPs is pretty damn "natural". To answer your questions:

  1. You would usually train the network's weights in random minibatches of data using a process called stochastic gradient descent (or some variant such as Adam, RMSprop, Adagrad etc...), rather than general gradient descent over the entire dataset at once. This is because each epoch is too expensive to calculate the gradients of weights over all of the data, vs a smaller random sample. Neural nets are trained iteratively and therefore kind of do improve through time (to a point, then they start overfitting). It is also common not to just run over the data once to get a decent fit, often you must run through a few times (each run through of data is called an "epoch"). Unlike Bayesian you don't really have a prior as the weights are usually randomly initialized, but the model (usually) keeps improving with more data / observations. There are Bayesian approaches to neural networks where you assume that the weights come from a distribution (say Gaussian prior over the weight space), then you can use MCMC - I don't think this is very popular anymore though.

  2. You can keep training by iterating over batches of different training data. In fact when data sets are large you basically have to break them up into different batches. So yes you can keep training them with different bits of data. Once some new data comes in you can then continue training, presumably you have saved the weights / model parameters. This won't start you from scratch again obviously.

I think if you code a network up or even just start with some Keras or Tensorflow implementation (on MNIST) you'll get a feel for what's going on and how they learn / behave in practice.

| improve this answer | |
  • $\begingroup$ What I wanted to say is that when you have a cost function $J(\theta)$ that you can split into pieces e.g. $J(\theta) = \sum\limits_{i=1}^m J_{i}(\theta)$ and you will minimize each piece separately then it is not necessarily the minimum of the whole thing $J(\theta)$. The parameters $\theta$ that you will find will only minimize the last part $J_{m}(\theta)$. Maybe the technique you described by taking random batches and re-runing everything many times truly find the minimum of the whole thing - like you are getting towards equilibrium. I should read more about this stuff. $\endgroup$ – Bociek May 4 '17 at 14:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.