1
$\begingroup$

I have ML ready samples. And each sample has a weight.

The weights distribute between [0-1]

My problem arise because there are a lot of samples which are 0.001, 0.00x And a lot of samples which are 0.997, 0.99x

I am going to sample data based on these weights. And samples with 0.99x will overshadow the other samples in the data set while 0.00x samples will have 0 significance.

The solution I am looking for, is some kind of function over those weights that will balance them a little bit / reduce those huge gaps (Therefor reducing the variance) AND still preserve their order

So if 0.997 turned into 0.88, 0.996 will turn into something < 0.88

For example:

in [0.01, 0.1, 0.4, 0.6, 0.8, 0.997]

I would want something like:

[0.15, 0.2, 0.42, 0.6, 0.78, 0.9] (Just an example)

$\endgroup$
1
  • $\begingroup$ I think I'm looking for distribution transformation $\endgroup$
    – Eran Moshe
    Apr 29, 2021 at 9:42

1 Answer 1

1
$\begingroup$

Try the softmax function:

weights = [0.01, 0.1, 0.4, 0.6, 0.8, 0.997]

temperature = 1
weights = np.array(weights) / temperature
new_weights = np.exp(weights) / np.exp(weights).sum() # softmax function

You can tweak the temperature hyperparameter. Higher the value - the "new_weights" come closer - yet order is preserved

$\endgroup$
6
  • $\begingroup$ @eran_moshe did that help? $\endgroup$ May 1, 2021 at 11:30
  • $\begingroup$ It's the direction I'm looking for.. Need to test it a bit and see the distribution change.. Maybe trying to normalize the distributions of those numbers will help ? $\endgroup$
    – Eran Moshe
    May 2, 2021 at 4:42
  • $\begingroup$ Yes, tweaking the temeprature will give you the desired level of normalization. $\endgroup$ May 2, 2021 at 4:53
  • $\begingroup$ That's a nice trick, this "Exponential smoothing" I will use it with a temperature of 0.25 It doesn't really normalize the data (and it probably shouldn't) But it does, effectively, reduce the relative gaps between the samples. How does this trick called in the field ? $\endgroup$
    – Eran Moshe
    May 2, 2021 at 5:54
  • 1
    $\begingroup$ never knew it could be used to reduce variance like this.. that "temperature" trick is very nice $\endgroup$
    – Eran Moshe
    May 2, 2021 at 6:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.