I am new to Data Science and have a few trivial questions, which I think are essential for my understanding of the basic data science techniques.
I am building a function to calculate a social welfare score/rank of the countries in the world. While doing so, I come across multiple outliers which essentially skew the results.
I have a few questions:
- Which normalization function to use and when? I am currently using the z score.
- If i come across an outlier and I know the reason for the outlier and want to avoid it affecting the result, how should I modify the value?eg . replace with mean/median. Which technique should I use and when?
- After calculating the z-scores, how to devise a function of the z scores? Is it trial and error based or I can apply some technique to find the best results. Since I do not have a strict result that I am expecting, how do I calculate the coefficients and operations for the z-scores?