Appropriate objective function and evaluation metric when I DO care about outliers?

Question

I am reading these two pages: xgboost documentation Post on evaluation metrics

I have a dataset where I am trying to predict future spend at the user level. A lot of our spend comes from large spenders, outliers. So, we care about them. I am using XGBoost.

I have tried xgboost with objective reg:squarederror. This tended to underpredict a little. I then tried with reg:squaredlogerror and this resulted in predictions that under predict by much more than just using squarederror.

I have tried tuning with several differing hyper parameter combinations but none made as big a difference as changing the objective. So, I'm dwelling on the objective function and trying to understand if there's another one out there that would be worth a shot?

On the xgboost docs above, some of the out of the other regression objective options are reg:pseudohubererror as well as count:poisson.

There is no option, that I can see, for just MAE. If using an objective function less susceptible to outliers with rmsle took me further away from accuracy whereas rmse took me closer, would using MAE potentially be worth a shot? In this dataset, outliers are more important, but so are regular users.

What would be a good objective and evaluation metric? Is MAE worth trying? If so, how? Looking at the docs above, I cannot see MAE as an option under regression parameters.

Answer 1

These are several things you can try:

• Use quartic error, $(y - \hat{y})^4$, instead of quadratic error. This is going to penalize a lot big errors, way more than MSE. The issue is that this is not implemented in xgboost, and you would need to develop a custom loss.
• If your target is always positive, you can use the target as training weights. This will give more weights to the outliers. If it is not always positive, you can use the absolute value of the target as weights. If using the target values directly puts too much weight on the outliers, you might want to transform it (e.g. using the log or square root), and if you have samples whose target value is zero, you might want to add some epsilon to all the weights. Note that xgboost can be easily trained using weights.
• Try to predict the quantile of the training distribution, then transform your predictions using the training cumulative probability function.

Answer 2

How to use least squares with weight matrix - use weights

from matplotlib import pyplot as plt
import numpy as np

# generate random data
N = 25
xp = [-5.0, 5.0]
x = np.random.uniform(xp[0],xp[1],(N,1))
e = 2*np.random.randn(N,1)
y = 2*x+e
w = np.ones(N)

# make the 3rd one outlier
y[2] += 30.0
w[2] = 0.0

######################################

from sklearn.linear_model import LinearRegression

# fit WLS using sample_weights
WLS = LinearRegression()
WLS.set_params(**{
    'copy_X': False,
    'fit_intercept': False,
    'n_jobs': -1,
    'positive': False
})
WLS.fit(x, y, sample_weight=w)
print(WLS.coef_)

# print(' residual sum of squares is : '+ str(np.sum(np.square(df['Predicted'] - df['Actual']))))
    
# for Lin.Regr. no param tuning, can use SGDRegressor instead if need to tune estimator's params

###################################### to check

import statsmodels.api as sm
mod_wls = sm.WLS(y, x, weights=w)
res = mod_wls.fit()
print(res.params)

# same result

############################# Plot

plt.plot(x, y, 'b.',  x, [email protected], 'g-', xp, xp*WLS.coef_[0], 'r-')
plt.legend(
           labels  = ['dots',  'statsmodels', 'LR'], fontsize="x-large")
plt.show()

but a little differ from SVM.SVR:

# ....... add previous code
from sklearn.svm import SVR
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline

from sklearn.utils.class_weight import compute_sample_weight
w= compute_sample_weight(class_weight='balanced', y=y)

# parameters
svr =  SVR(kernel='linear',  tol= 1.3e-10, epsilon = 0.01)
svr.fit(x, y.ravel(), sample_weight=w)
y_pred = svr.predict(x)
print("SVR: ", svr.coef_)

pipeline = Pipeline( [ ('r', SVR(kernel='linear',  tol= 1.3e-6, epsilon = 0.0001) ), ] )
##print(pipeline.get_params().keys())
##y_pred = pipeline.fit(x, y).predict(x)
##print("SVR: ", pipeline['r'].coef_)

grid = GridSearchCV(pipeline, param_grid={"r__C":[1,5,10],"r__gamma":  [1e-8, 1e-7, 1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1]}, cv = 5, n_jobs=-1, verbose=2)
grid.fit(x, y.ravel(), r__sample_weight=w)
print("GridSearchCV: ", grid.best_estimator_.steps[-1][1].coef_)
y_grid = grid.predict(x)

# Plotting
plt.plot(x, y, 'b.',  x, [email protected], 'g-', x, y_pred, 'r-', x, y_grid, 'k-')
plt.legend(
           labels  = ['dots', 'statsmodels',  'SVR', 'SVR_CV'], fontsize="x-large")
plt.show()

res:

LinearRegression: [[2.02717863]]
statsmodels: [2.02717863]
SVR: [[1.96308977]]
SVR_with_GridSearchCV: [[1.96308977]]

for SVR "outliers will still affect" - though can make use of the epsilon-insensitive loss - but seems not very helpfull

p.s. Supervised ML cheat-sheet