3
$\begingroup$

I have a data set of with a distribution that looks something like this:

enter image description here

I need to take random sample data from the set so that distribution will be more even. Something like this (take the data in the green area):

enter image description here

I know how to do this by taking the data, putting into separate "buckets" (distribute the data into X buckets, taking up to Y samples from each bucket), but I was wondering if there is there an easier way.

P.S. the result doesn't have to be 100% accurate - a good approximation is enough.

$\endgroup$
2
  • $\begingroup$ The buckets solution will be approximate and good enough. Why do you need a uniform distribution? $\endgroup$
    – K3---rnc
    Nov 29 '16 at 16:52
  • 1
    $\begingroup$ I implemented the buckets solution it's huge amounts of data and the sorting into buckets takes a long time. Also, I'm curious if there's a more "mathematical" solution. $\endgroup$
    – traveh
    Nov 30 '16 at 8:36
4
+50
$\begingroup$

There are a variety of sampling options:

  1. Sample directly from the data which perfectly models the empirical distribution.
  2. Fit a kernel density estimation (kde). Sample from the estimated kde function.
  3. Create a histogram of the data, aka bin the data. Then treat each histogram as a probability mass function (pmf). Sample from bins proportional to their frequency.

You can create variations of the data or distributions:

  1. Apply a transformation to the data. For example, a log transformation will transform skewed distribution to be more normal.
  2. The histogram values could be changed to any shape.

Then sample from the changed data or distributions.

The most extreme option would define a uniform distribution across the data domain and sample from that distribution. The green box is a uniform distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.