1
$\begingroup$

In my scenario, I have to process some input data and give a score based on what the processing phase outputs. The problem is that, in order to scale the score in a human-readable format I'd have to know what a possible maximum value could be.

As there is no knowledge about what a maximum value could be in reality, I have chosen to scale the scores based on the current existing maximum value. This approach can work well but is really disturbed by some obvious outliers.

My question is: what would be a more intelligent way to scale a given data instance besides picking a threshold that represents a maximum?

More context about the data: a series of information is received about a certain identifier, on which there are made checks for certain conditions. Long-story short, in this step it is just about counting appearances and multiplying them with individual weights, that add up to a score. One can think about this score as a multi-variable function.

This score is wished to be scaled so that it represents something more tangible to a non-expert reader. Instead of receiving the value of '239', one receives the value of '7.8', scaled in the interval of 1-10. In this way, by knowing that the maximum readable value is 10 and that you have a score of 7.8, you can make clear assumptions of your situation.

The data is not tied to a given point in time, but rather to an input that changes over time. Therefore, the values are quite dynamic and scaling them in the min/max way would give different results in different points of time, even though nothing changed about a particular instance (it just happened that a new instance appeared, that has a greater score).

Improve this question
$\endgroup$
5
  • $\begingroup$ Tells us more about the context and data. Is this time series data? Why exactly do you want to scale your results? What is this score? $\endgroup$
    – user2974951
    Jun 14 at 11:51
  • 1
    $\begingroup$ I have added more explanations. Hope it is more clear now. $\endgroup$
    – Ionut-Alexandru Baltariu
    Jun 14 at 11:57
  • $\begingroup$ @user2974951, is the provided data enough? I'm still looking for other possibilities of scaling. $\endgroup$
    – Ionut-Alexandru Baltariu
    Jun 20 at 10:54
  • $\begingroup$ You say that the data is not tied to a given point in time, and that the input changes over time. Does that mean that your min and max features should also change over time? I.e. there are periods of higher volatility and lower volatility, and you want to take this into account? Some sort of rolling relative windows? Or a rolling absolute range, where relative changes do not matter but only the actual size? $\endgroup$
    – user2974951
    Jun 20 at 12:44
  • $\begingroup$ Maximum value can change over time when the input contains a value that is greater than the current maximum value. The minimum value will always be 0. $\endgroup$
    – Ionut-Alexandru Baltariu
    Jun 20 at 14:19

2 Answers 2

Reset to default
1
$\begingroup$

You can choose a different normalization process instead of standard min-max scaler.

If your input data are representative of the actual data, you can do a z-score using a linear mapping as y = (x-m) / s where m is the mean of your data and s is its standard deviation. You can find that as standard scaling as well. MinMax Scaling can be really problematic if there are outliers.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Standardization will just as well be problematic with outliers. $\endgroup$
    – user2974951
    Jun 14 at 11:47
0
$\begingroup$

My suggestion is to use an empirical cumulative distribution function and score new points based on this. This will result in a percentile which will always be in [0,1], and is more robust to outliers. You can then multiply/add something to this to make it on a [1,10] scale, although I would use a [0,10] scale, which you can obtain simply by multiplying the percentiles by 10.

Some sample code in R

> x=c(0,1,1,2,3,5,8,13,21,1000)
> ecdf(x)(0)
[1] 0.1
> ecdf(x)(5)
[1] 0.6
> ecdf(x)(34)
[1] 0.9
> ecdf(x)(1001)
[1] 1
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.