what is the meaning of independent and identically distribution of samples in a dataset for neuralnetworks

Question

random variables like heads or tail that generated by flipping a coin is independent because each time we tossing the result isn't depend on previous toss( in other words the function that generates random variable doesn't have memory). the example also is identically distributed because each toss result follows same distribution (binomial distribution).

but what is the meaning of IID in the case of images classification( like hand written digits dataset). each sample is independent from previous samples but how we ensure they generated with the same probability distribution(or what is the meaning of that in this specific case)?

or in the case of recurrent neural network after that we create over sample (according to the constrains of the problem) from the series. we can do shuffling to ensure the samples in dataset are independent but how about identically distribution of samples?

Answer 1

Once the data is with us, the best we can do is shuffling the samples to make them Identically distributed assuming a hypothetical process behind it that generates all the samples.

Let's take the example of MNIST digits, the hypothetical process is our similar knowledge of digits, so most of us will write a specific digit in a similar way.

Now let's test the MNIST model on Japanese MNIST, it will fail as the assumption is broken.

Again, mix both the MNIST, Shuffle it, Split it and Train it. It will learn the pattern and can predict the digits.
Once again, both the train/test have a similar probability distribution of pixels.

While training the model,
If the process is incremental e.g. Gradient Descent, we try to assure that each batch represents the parent distribution. Again this is achieved by Shuffling esp. Stratified shuffling.