Well, after years, now I really know why we shuffle data! The idea is very simple, but I do not know why we really did not consider it.

For making the cost function, we are explicitly considering that the samples are i.i.d. For instance, in binary cross-entropy, you can easily see that we have a summation. That summation has been a product at first, and after taking the logarithm, it has been changed to sum. Actually, in the formulation of that cost function, we have discarded the joint probability, because it is difficult to compute. With i.i.d assumption, we have the current cost function.

Well, after years, now I really know why we shuffle data! The idea is very simple, but I do not know why we really did not consider it.

For making the cost function, we are explicitly considering that the samples are i.i.d. For instance, in binary cross-entropy, you can easily see that we have a summation. That summation has been a product at first, and after taking the logarithm, it has been changed to sum. Actually, in the formulation of that cost function, we have discarded the joint probability, because it is difficult to compute. With i.i.d assumption, we have the current cost function. Now suppose our task is learning with different mini-batches and these mini-batches are not identical.