This question is part of a sample exam that I'm working on to prepare for the real one. I've been stuck on this one for quite a while and can't really motivate my answers since it keeps referring to guarantees, can someone explain what guarantees we can have in a neural network such as the one described?
Assume a deep neural network (DNN) that takes as input images images of handwritten digits (e.g., MNIST). The input network consists of 28x28 = 784 input units, a hidden layer of logistic units, and a softmax group of 10 units as the output layer. The loss function is the cross-entropy. We have many training cases and we always compute our weight update based on the entire training set, using the error backpropagation algorithm. We use a learning rate thats small enough for all practical purposes, but not so small that the network doesnt learn. We stop when the weight update becomes zero. For each of the following questions, answer yes or no, and explain very briefly.
Will this DNN configuration enable the the weights to minimize the loss value (there may be multiple global optima)?
Does this DNN configuration guarantee that the loss reduces on every step? We use our network to classify images by simply seeing which of the 10 output units gets the largest probability when the network is presented with the image, and declaring the number of that output unit to be the networks guessed label.
Does this DNN configuration guarantee that the number of mistakes that the network makes on training data (by following this classification strategy) never increases?