2
$\begingroup$

I used cross validation on my data (11000 rows) with maximum salary of 10000 and after some cleaning I got to rmse=70. Then I tried to remove the outliers 10 times just to try things now I have 9000 rows with maximum salary of 260, I got rmse=23. Is what I did bad even though I got a better rmse? Is the jump from 10000 maximum to 260 a bad thing? Is the jump from 11000 rows to 9000 a bad thing?

The Above one is before removing the outliers and the one under is after removing them

$\endgroup$
8
  • $\begingroup$ That depends on the statistics of your data. Other than max, you should also share min, std and possibly 25%, 50%, 75% quantiles. $\endgroup$
    – serali
    Commented Oct 4, 2021 at 11:36
  • $\begingroup$ @serali I edited the question and added them (The above one is before removing the outliers and the under one is after removing them) $\endgroup$
    – Maxi
    Commented Oct 4, 2021 at 11:41
  • $\begingroup$ For which dataset have you calculated the rmse? Train or test/validation? $\endgroup$
    – spectre
    Commented Oct 4, 2021 at 11:41
  • $\begingroup$ @spectre can you ask your question in other words? I used the cross validation with cv = 10 $\endgroup$
    – Maxi
    Commented Oct 4, 2021 at 11:48
  • $\begingroup$ You've accepted a problematic answer. This is one of the reasons why it is suggested not to accept an answer for a few days. $\endgroup$
    – Dave
    Commented Oct 4, 2021 at 14:57

2 Answers 2

5
$\begingroup$

Removing outliers is only appropriate when you have reason to believe the data is wrong. Do you have such a reason? Otherwise, you are, as @Dave suggested, tricking yourself into thinking you have good predictive power.

If your data is not "nicely" distributed, and you're having trouble fitting a model to predict it, the first thing I would try is transform the salary field to a more usable range. For example, you can try predicting log(salary) or sqrt(salary), then transforming it back if necessary.

$\endgroup$
-1
$\begingroup$

Your data is distributed very strangely, to say the least. A mean value of 178, stdev 225 and %75 is 176 is probably as far from a normal distribution as possible. But it is probably a good idea to get rid of anything out of 3 standard deviation from your mean value, that is 178 + 3*225 ~ 853.

There are different ways to deal with skewed data, but for your case it is probably best to get rid of some outliers. But as @Dave points below, this lowers the predictive power for extreme values, so it is more of a judgement call.

$\endgroup$
4
  • $\begingroup$ Thank you very much for the clear answer, I'll try the things you mentioned. $\endgroup$
    – Maxi
    Commented Oct 4, 2021 at 12:19
  • $\begingroup$ @FjkgB revised the normalization link to a better one, previous one was not really appropriate for this skewed data. $\endgroup$
    – serali
    Commented Oct 4, 2021 at 13:36
  • 1
    $\begingroup$ -1 1) We don't consider the normality of the marginal variable, and when we consider the normality of the error term (estimated by the residuals), we tend to do so for reasons of inference rather than prediction. 2) Outlier removal is fraught with problems. In this case, I would say that if you remove the outliers and find yourself making better predictions, you're only tricking yourself into thinking that you have good predictive power, but you do not have a way to handle the extreme measurements. $\endgroup$
    – Dave
    Commented Oct 4, 2021 at 14:56
  • $\begingroup$ @Dave you should probably post this as a separate answer, OP asking whether he did good or bad. $\endgroup$
    – serali
    Commented Oct 4, 2021 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.