I am buliding a machine learning model with logistic regression. I am dealing with blood transfusion data set. In which there is a feature,Total_volume, I found that there is more than 5% of outliers present.Should I handle this? Or I keep this data as it is? Bacauase more than 5% data may contain some patterns.
3
-
$\begingroup$ How did you find these "outliers"? $\endgroup$– user2974951Aug 11, 2022 at 6:33
-
$\begingroup$ The answer to this questions, among other things, is highly domain dependent. Is it normal to have such values in blood transfusion, could it just be an anamoly ? $\endgroup$– PolymathAug 11, 2022 at 6:50
-
$\begingroup$ I drew boxplot for visualizing the outliers. $\endgroup$– Neenu PrasadAug 11, 2022 at 8:10
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
I'm assuming you've classified outliers using some rule, like over 3 standard deviations or something?
Look at a histogram of your data; is it normally distributed with outliers? Or is it non normally distributed? If you run normality tests on data that just isn't normal (e.g. reaction time, or income) then you will think you have lots of outliers when you don't.
If your data isn't normally distributed there are lots of ways to address it, including transforming the data (e.g. a log transform) or by....ignoring it!
Also, if the 'outliers' were definitely mistaken values, perhaps it's ok to remove them, but if they are normal values, remove them with caution, because your model will still encounter these individuals when it is deployed, so removing them now could mean worse performance in production.
$\begingroup$
$\endgroup$
I have worked for Bank and e commerce . The outliers can be set at as low as .1% or as high as 20%.
- Plot a scatter curve or box plot
- Start with hypothesis that 0% are outliers
- Determine how many points you are excluding from dataset by removing next 1%.
If the exclusion is significant, then that is your outliers %
