Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am revising undergrad statistics course via this course, where i am learning techniques to pull out sample from population.
While ensuring that sample is a decent representative of the population, I am left with a question.
Here is my work from where i came up with this question
It depends on how you want to treat outliers. Outliers can either be generated by chance from the data's distribution or can be a result of human error (error in measurement, error in data entry, etc.).
If you consider the outlier to be from the first category, in my opinion you shouldn't discard it, in order to take your samples from the true distribution of the data.
Outliers in this category, might indicate a heavily-skewed distribution, or might simply appear due to the nature of the problem (e.g. King's effect).
If you consider it to be an error (e.g. a human height of 3.75m), you should discard it as it corrupts the distribution of the data.
In order to determine which category the outliers belongs to, either requires domain knowledge (e.g. knowing that human height can't exceed 3m) or requires you to make assumptions on the data.
