# Reducing (Variance) | the gap between my weights

I have ML ready samples. And each sample has a weight.

The weights distribute between [0-1]

My problem arise because there are a lot of samples which are 0.001, 0.00x And a lot of samples which are 0.997, 0.99x

I am going to sample data based on these weights. And samples with 0.99x will overshadow the other samples in the data set while 0.00x samples will have 0 significance.

The solution I am looking for, is some kind of function over those weights that will balance them a little bit / reduce those huge gaps (Therefor reducing the variance) AND still preserve their order

So if 0.997 turned into 0.88, 0.996 will turn into something < 0.88


For example:

in [0.01, 0.1, 0.4, 0.6, 0.8, 0.997]

I would want something like:

[0.15, 0.2, 0.42, 0.6, 0.78, 0.9] (Just an example)

• I think I'm looking for distribution transformation – Eran Moshe Apr 29 at 9:42

## 1 Answer

Try the softmax function:

weights = [0.01, 0.1, 0.4, 0.6, 0.8, 0.997]

temperature = 1
weights = np.array(weights) / temperature
new_weights = np.exp(weights) / np.exp(weights).sum() # softmax function


You can tweak the temperature hyperparameter. Higher the value - the "new_weights" come closer - yet order is preserved

• @eran_moshe did that help? – Jayaram Iyer May 1 at 11:30
• It's the direction I'm looking for.. Need to test it a bit and see the distribution change.. Maybe trying to normalize the distributions of those numbers will help ? – Eran Moshe May 2 at 4:42
• Yes, tweaking the temeprature will give you the desired level of normalization. – Jayaram Iyer May 2 at 4:53
• That's a nice trick, this "Exponential smoothing" I will use it with a temperature of 0.25 It doesn't really normalize the data (and it probably shouldn't) But it does, effectively, reduce the relative gaps between the samples. How does this trick called in the field ? – Eran Moshe May 2 at 5:54
• never knew it could be used to reduce variance like this.. that "temperature" trick is very nice – Eran Moshe May 2 at 6:59