How to shuffle input data using stochastic gradient decent?

Question

Let's say that I have input with size $n\times n$ and after that I operated shuffled operations and get the output with the same dimension. But I don't know if we can do that or not. The idea is to permute all the configuration and find the minimum configuration when given loss. But the operation became very expensive with the growth of $O((n^2)!)$, and with each iteration we need to compute loss separately. So is there an optimization for such problem or it just not feasible?

In other words, is there a gradient correlation between different configuration that will lead to the local minima of the loss function?

Answer 1

If I understand your question correctly, I would advise you not to spend time on this.

If you have data that has a structural order to it, for example time-series data where each sample follows from the previous, then shuffling would be throwing that information away.

If your data is not sequential, on the other hand, then computing all possible permutations and training the model on each of them is really just a waste of time. We shuffle the data e.g. to prevent a powerful model from trying to learn some sequence from the data, which doesn't exist.

Training a model on all permutations might be a way to uncover the correct order of the data, is you know it has an order, but it was already shuffled. Otherwise, this experiment would be as useful as training the same model $(n^2)!$ times, just using a different random seem each time.

Have a look here for some more points on shuffling data and why we might do it. Here is more discussion on shuffling input samples.