I was thinking if anyone considered a sampling technique that would try to aim keeping as much of the variance as possible (e.g. as many unique values, or very widely distributed continuous variables).

The benefit might be that it will allow development of code around the sample, and really work with the edge cases in the data.

You can then later always take a representative sample.

So, I am wondering if people have tried to sample for maximum variance before and if there is a clever way to sample with as high possible variance (of course an approximation is just fine).

  • $\begingroup$ I am now considering expanding categorical variables into dummies, computing z-scores over all columns (in any case), summing them so each row has a single score, and then if interested in a sample size of 10% of the original data, then to select the rows with the top 5% and bottom 5% scores. $\endgroup$ Sep 1 '19 at 9:22

It depends on what you mean by sampling. Is it sampling between or within features?

For between features, scikit-learn has a built-in option for VarianceThreshold which removes features whose variance does not meet some threshold.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.