How can I identify and remove outliers in R

Question

I am performing regression analysis on prices of product that we have purchased, based on size and other attributes.

However there are often buys in odd circumstances which factor into the price, that is not (and cannot be) addressed directly in the features of the analysis.

Each time I run a regression, I will check the 20 with the largest error manually, and 90%+ of the time they will be odd buys like mentioned before, and for my purposes can be completely ignored.

I have been looking into cooks distance to remove these, however I'm not sure how to best set the threshold, or if there is a better method to use.

Answer 1

Cook's distance and the alternative method DFFITS are not strictly speaking methods to detect 'outliers' in the sense of purely anomalous values, rather they detect 'influential' points, where leaving the value in or out of the analysis noticeably changes the result. Hence, Cook's distance measures how much the beta values change when a specific observation is omitted, which is a different idea to detecting an outlier as such, although they will naturally often be the same points.

From the description of your problem, it is possible that detecting influential points is what you are actually trying to do.

To directly answer your question on the Cook's distance threshold, the F-statistic with p and n-p (where p is number of regressors, and n is observations) degrees of freedom is often used.

Outliers per se in regression are more often identified via residual analysis e.g. if the data point's corresponding residual is large in the context of the data set. This is often taken to be when the value lies more than three standard deviations away from the mean of the residuals (though note that in a sufficiently large data set, a well fitted model will still have some observations at this level)

Answer 2

You should use robust regression e.g. lmrob from R-package robustbase.