I have a general question on reservoir sampling. When I use this method to sample a very large dataset for training machine learning classification algorithms, I am curious as to how to make my pipeline robust to fluctuations in class distribution across samples.
For example, suppose I am working on a binary classification problem. I want to sample a relatively small subset of data on which to evaluate my algorithms, so I used reservoir sampling. However, it seems to me (thought I could be wrong) that by chance I may draw a sample wherein the class distribution is significantly different from that seen in the larger population of data. If this is a correct inference, can anybody tell me how to remedy the situation? If reservoir sampling is not the answer, what other procedure are out there that I can leverage in this case (apart from using distributed environment)?
I am learning, so some explanation, hint, and direction would be greatly appreciated.