0
$\begingroup$

I am buliding a machine learning model with logistic regression. I am dealing with blood transfusion data set. In which there is a feature,Total_volume, I found that there is more than 5% of outliers present.Should I handle this? Or I keep this data as it is? Bacauase more than 5% data may contain some patterns.

$\endgroup$
3
  • $\begingroup$ How did you find these "outliers"? $\endgroup$ Aug 11, 2022 at 6:33
  • $\begingroup$ The answer to this questions, among other things, is highly domain dependent. Is it normal to have such values in blood transfusion, could it just be an anamoly ? $\endgroup$
    – Polymath
    Aug 11, 2022 at 6:50
  • $\begingroup$ I drew boxplot for visualizing the outliers. $\endgroup$ Aug 11, 2022 at 8:10

2 Answers 2

1
$\begingroup$

I'm assuming you've classified outliers using some rule, like over 3 standard deviations or something?

Look at a histogram of your data; is it normally distributed with outliers? Or is it non normally distributed? If you run normality tests on data that just isn't normal (e.g. reaction time, or income) then you will think you have lots of outliers when you don't.

If your data isn't normally distributed there are lots of ways to address it, including transforming the data (e.g. a log transform) or by....ignoring it!

Also, if the 'outliers' were definitely mistaken values, perhaps it's ok to remove them, but if they are normal values, remove them with caution, because your model will still encounter these individuals when it is deployed, so removing them now could mean worse performance in production.

$\endgroup$
0
$\begingroup$

I have worked for Bank and e commerce . The outliers can be set at as low as .1% or as high as 20%.

  1. Plot a scatter curve or box plot
  2. Start with hypothesis that 0% are outliers
  3. Determine how many points you are excluding from dataset by removing next 1%.

If the exclusion is significant, then that is your outliers %

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.