My data set is of shape (60,784,1000) with mini batches for input and (60,10,1000) for labels, should I shuffle only the 60 mini batches or the training examples themselves?

  • $\begingroup$ could you tell what these dimensions represent? Usually, the first dimension is used for the number of observations, it does't seem to be the case here. How did you organize your dataset? $\endgroup$
    – Leevo
    Apr 27 '20 at 21:08
  • $\begingroup$ In this case the 60 represents the number of mini batches, 784 and 10 is the input and output size, and 1000 the number of training examples per batch $\endgroup$ Apr 27 '20 at 21:13

Normally, you would shuffle up all of the examples and then portion them off into batches of some chosen size. Then, you would do a parameter update based on the gradient of the loss function with respect to each batch. This whole process is one "epoch" of training. Typically, deep neural nets are then trained over many epochs, often with a learning rate that varies as training proceeds.

An important aspect of this process is that when the data is shuffled up at the beginning of an epoch, examples are put into batches with different examples than they were matched with in the previous epoch. This gives us a more complete sampling of batch gradients and improves our collective stochastic estimation of the optimal gradient (the derivative of the cost function with respect to the model parameters and data).

Short answer: your model performance will almost certainly be worse if you choose static batches and shuffle those batches around instead of shuffling the data and then dividing them into batches.

Also, be careful of your shapes. With a dataset shape of (60, 784, 1000), it's highly likely that you're working on MNIST or one of its cousins - MNIST is 60,000 examples of length (784,), if the images have been flattened down from 28 x 28 pixels. The 784 being axis 1 in your shape tuple makes me concerned that you've reshaped your data incorrectly. Make sure that the array entries that represent a single image are where you think they are.

  • $\begingroup$ I see! Thank you so much, in depth complete answer, I appreciate it very much, the warning about the data shape was also good, I will check it immediately. $\endgroup$ Apr 27 '20 at 21:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.