I build xgboost model for regression problem. By the default xgboost optimize $(y - y_{pred})^2$, so the RMSE will be the best eval metric to measure performance. But my task is to build the best model for evaluation metric which check if predicted value is in range $-/+10%$ of true value i.e. $y_{pred} \in [0.9*y,1.1*y]$. Do you have any idea for this or maybe MSE is still the best option?
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
In my experience, it is very difficult to come up with good working custom objective functions for xgboost.
Custom objectives need to be continuous and need to have a convex gradient and non zero hessian, which is often not the case for custom loss functions.
One simpler method you could use is to define a custom validation metric based on your range +/-10, that can be used in conjunction with early stopping to optimize your hyperparameters of the xgboost model. So the model will not directly optimize for this, but will select hyperparamters that will minimize your custom loss. However, I would suggest to stick with the RMSE objective for this problem.
$\begingroup$
$\endgroup$
You can have two approaches:
- As mentioned by @fhaase, continue your regression model, until you get the desired output i.e. -/+ 10.
- You can also try it as a binary classification problem, where if the output is with in the desired range treat it as 1 otherwise 0. In that case you have a chance to explore different cost function like log_loss.
So steps will be:
- Try any regression algo.
- Apply Sigmoid function on top of the outcome of steps 1.
- Optimize the output of step 2.
