Suppose I have 1-D data which has some outliers, I want to normalize the data to be in the range [0,1]. I tried calculating the maximum value and the minimum value as follows:

q1,q2,q3 = quartiles of the data
max = q3 + (q3-q1)*1.5
min = q1 - (q3-q1)*1.5

I used the above approach because I have read that data above maximum or data less than the minimum (as calculated above is noise).

My question is: whatever I am doing, is it correct or is there any other way to achieve good results?

Thank you for helping.

  • $\begingroup$ please define normalization. This tag has no definition in data-science. $\endgroup$ Sep 13 '20 at 0:35

The question,

whatever I'm doing is it correct or ...

will occur over and over again, until you have a firm understanding of statistics. See if you are looking for short-term learning, then you might pick up some fragments of information here and there, but, for long term learning, I'll suggest that you read a good text on introductory statistics. Try, this book, or this one on Applied Predictive Modeling

Now to answer the question, Yes what you are doing is correct. I've explained it below;

Outliers are observations that fall below the lower limit or above the upper limit. The five number summary of a continuous variable (or numerical variable) is Min, Q1, Q2, Q3, Max. Where the lower and upper limits are defined as, Lower Limit = Q1 - 1.5 x IQR, Upper Limit = Q3 + 1.5 x IQR and IQR is the Inter Quartile Range, which is the difference between the first and third quartile. Q1 is the first quartile, Q2 is the second quartile or the median and Q3 is the third quartile.

Finally, please always plot your data.

  • $\begingroup$ I wholeheartedly would like to thank you. I will definitely give this a read. Can you please give me an answer as well? $\endgroup$ Sep 9 '20 at 15:04
  • $\begingroup$ @satindersingh I have revised the answer. Hope it helps. $\endgroup$
    – mnm
    Sep 9 '20 at 22:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.