2
$\begingroup$

I am new to Data Science and have a few trivial questions, which I think are essential for my understanding of the basic data science techniques.

I am building a function to calculate a social welfare score/rank of the countries in the world. While doing so, I come across multiple outliers which essentially skew the results.

I have a few questions:

  1. Which normalization function to use and when? I am currently using the z score.
  2. If i come across an outlier and I know the reason for the outlier and want to avoid it affecting the result, how should I modify the value?eg . replace with mean/median. Which technique should I use and when?
  3. After calculating the z-scores, how to devise a function of the z scores? Is it trial and error based or I can apply some technique to find the best results. Since I do not have a strict result that I am expecting, how do I calculate the coefficients and operations for the z-scores?
| improve this question | |
$\endgroup$
  • $\begingroup$ Can you illuminate what you mean by "function of the z-scores"? $\endgroup$ – grldsndrs Sep 24 '16 at 0:09
  • $\begingroup$ My dataset has a few columns like the GDP of the country, literacy rate, migration rate etc. I have calculated the z scores for each of them. My final score for the country could be a function of the calculated z-score. Can be a simple addition or a weighted addition. $\endgroup$ – siddshah Sep 24 '16 at 2:44
  • $\begingroup$ The transform: z = (X - μ) / σ , i.e. z is X normalized and scaled to zero-mean, unit-stdev. $\endgroup$ – smci Nov 23 '16 at 15:04
  • $\begingroup$ Is your classification function linear regression? or else what? "To calculate a social welfare score/rank" Presumably you want a regressor to estimate the score, then you calculate the ranks, right? $\endgroup$ – smci Nov 23 '16 at 15:07
2
$\begingroup$

Just some quick fixes. Normalization takes care of scale of data across the columns. e.g. If one data column is in the ranges of 1000s and other in 10s then normalization will work.

But if you want to take care of the skew-ness one way is to simply take log of the data. It pulls in the outliers. as log increases very slowly for positive values. Do it before the normalization (Z score in your case) enter image description here

Other ways are taking nth root of the data.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Which normalization function to use and when?

Normalization (statistics)

Which outlier technique should I use and when?

How to deal with outliers?

How to devise a function of the z scores?

I am not able to give any specific answer, but my strategy would be to try and find meaningful relationships with the features. I would then try and highlight those relationships with the statistics.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.