So I recently finished a mini batches algorithm for a library in building in java(artificial neural network lib). I then followed to train my network for an XOR problem in mini batches size of 2 or 3, for both I got worse accuracy to what I got from making it 1(which is basically just SGD). Now I understand that I need to train it on more epochs but I'm not noticing any speed up in runtime which from what I read should happen. Why is this?

Here is my code(Java)

 public void SGD(double[][] inputs,double[][] expected_outputs,int mini_batch_size,int epochs, boolean verbose){
    //Set verbose

    //Create training set
    TrainingSet trainingSet = new TrainingSet(inputs,expected_outputs);

    //Loop through Epochs
    for(int i = 0; i<epochs;i++){
        //Print Progress
        print("\rTrained: " + i + "/" + epochs);

        //Shuffle training set

        //Create the mini batches
        TrainingSet.Data[][] mini_batches = createMiniBatches(trainingSet,mini_batch_size);

        //Loop through mini batches
        for(int j = 0; j<mini_batches.length;j++){

    //Print Progress
    print("\rTrained: " + epochs + "/" + epochs);

   private Pair backprop(double[] inputs, double[] target_outputs){
    //Create Expected output column matrix
    Matrix EO = Matrix.fromArray(new double[][]{target_outputs});

    //Forward Propagate inputs

    //Get the Errors which is also the Bias Delta
    Matrix[] Errors = calculateError(EO);

    //Weight Delta Matrix
    Matrix[] dCdW = new Matrix[Errors.length];

    //Calculate the Deltas
    //Calculating the first Layers Delta
    dCdW[0] = Matrix.dot(Matrix.transpose(I),Errors[0]);

    //Rest of network
    for (int i = 1; i < Errors.length; i++) {
        dCdW[i] = Matrix.dot(Matrix.transpose(H[i - 1]), Errors[i]);

    return new Pair(dCdW,Errors);
private void update_mini_batch(TrainingSet.Data[] mini_batch){
    //Get first deltas
    Pair deltas = backprop(mini_batch[0].input,mini_batch[0].output);

    //Loop through mini batch and sum the deltas
    for(int i = 1; i< mini_batch.length;i++){

    //Multiply deltas by the learning rate
    //and divide by the mini batch size to get
    //the mean of the deltas

    //Update Weights and Biases
    for(int i= 0; i<W.length;i++){
  • $\begingroup$ did you try it for different sizes of training sets? if training sets are too small you may not notice the difference $\endgroup$
    – Javi
    Mar 6, 2019 at 17:06
  • $\begingroup$ Well the training set itself is 60k for the mnist dataset, and I did mini batches of 300 $\endgroup$ Mar 6, 2019 at 20:46

1 Answer 1


My understanding is that mini-batches are not really for speeding up the calculations... but to actually allow large datasets to be calculated.

If you have 1,000,000 examples, it would be tricky for a computer to compute forward and backward passes, but passing batches of 5,000 elements would be more feasible.

For your case, I recommend you two things

  1. Try different batch sizes.
  2. Make sure you shuffle your batches!!! that will certainly help you a bit.
  • $\begingroup$ Alright, I mean the XOR problem has only 4 possible inputs so that might be why batches may be slower in this case. I also have shuffling implemented already. Lastly, When i try to train on a large dataset like MNIST dataset, the network doesn't learn at all from my experience with this library. I posted another issue if you think you could help me. datascience.stackexchange.com/questions/46651/… $\endgroup$ Mar 7, 2019 at 19:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.