For outliers treatment: clipping, winsorizing or removing?

Question

I came across three different techniques for treating outliers winsorization, clipping and removing:

• Winsorizing: Consider the data set consisting of: {92, 19, 101, 58, 1053, 91, 26, 78, 10, 13, −40, 101, 86, 85, 15, 89, 89, 28, −5, 41} (N = 20, mean = 101.5) The data below the 5th percentile lies between −40 and −5, while the data above the 95th percentile lies between 101 and 1053. (Values shown in bold.) Then a 90% winsorization would result in the following: {92, 19, 101, 58, 101, 91, 26, 78, 10, 13, −5, 101, 86, 85, 15, 89, 89, 28, −5, 41} (N = 20, mean = 55.65)

• Clipping:Given an interval, values outside the interval are clipped to the interval edges. For example, if an interval of [0, 1] is specified, values smaller than 0 become 0, and values larger than 1 become 1.

• Removing: Just taking them out.

My questions are:

1. In which cases should I use which one?
2. If I always do winsorization (which seems the best in my opinion) when I am losing important information?
3. Is this model-dependent (for decision trees, for linear...) or the same strategy can be applied to all of them

Answer 1

On the difference between winsorizing and clipping :

• The techniques are very similar. They deal with extreme values (that are not necessarily outliers). Imo you should generally avoid thinking that big values = outliers. Solutions to deal with big values include normalizing your variables by a size factor for more comparability.
• Which one should you use is linked to your data and their predictiveness. Sometimes data outside of a threshold will be predictive, sometimes not. Sometimes the underlying process will depend on rank, sometimes on global values. There is no rule for when to do one of those two techniques, there is no rule on which one to choose. In the end, you should choose which one gives you better results. Yeah, windsorisation seems a bit more 'adaptative' as you don't have to look at each of your predictors to choose a threshold, but outside of that, there is no reason it performs better statistically.
• One of the main problems of those techniques is that they are univariate. If your predictors are correlated, modifying one value without another can breaks your relationships.
• Yes, the answer is model dependent. Having extreme value may be detrimental for the convergence of methods based on gradient descent, as some data points may swing the optimization path. For methods relying on the partition of the space like trees, having extreme values is less of a problem.

Removing outliers is a completely different strategy. The difficulty is to identify outliers confidently... but that's another question.