Hyperparameter tuning XGBoost

Question

I'm trying to tune hyperparameters with bayesian optimization. It is a regression problem with the objective function: objective = 'reg:squaredlogerror'
$\frac{1}{2}[log(pred+1)-log(true+1)]^2$

My dataset consists of 20k vectors, each vector has length 12 (twelve features). Every vector has a corresponding Y value.

I want to find the set of hyperparameters that minimize the loss function. This is how it is implemented in code:

def evaluate_model(learning_rate, max_depth, nr_estimators, min_child_weight, min_split_loss, reg_lambda):
    model = get_model(learning_rate, max_depth, nr_estimators, min_child_weight, min_split_loss, reg_lambda)
    model.fit(X_train, Y_train)
    pred = model.predict(X_val)

    error = np.array([])
    for i in range(len(pred)):
        prediction = np.maximum(pred[i],1)
        error = np.append(error, (1/2)*(np.log(prediction+1)-np.log(Y_val[i]+1))**2)
    err = np.mean(error)
    return -err

My question is if anyone has any problem with how I've constructed the evaluate_model function. Do this optimize the squared log error when bayesian hyperoptimization is being implemented? The maximum(pred[i],1) is there in case a negative prediction is produced. Also, I get bad results even after the hyperparameter optimization.

These are the hyperparameters I evaluate:

pbounds = {'learning_rate': (0,1), 'max_depth': (3,10), 'nr_estimators': (100, 5000), 'min_child_weight': (1,9), 'min_split_loss': (0,10), 'reg_lambda': (1,10)}

The optimization is ran for 100 iterations and 10 init points. The package I've used for the bayesian optimization is bayes_opt

Answer 1

Another way is to use the mean_squared_log_error from the same metrics module,.

First clip the negative values in the predictions to 1 and find the mean squared log error

pred = np.clip(pred, min=1, max=None)

err = mean_squared_log_error(yval, pred)

Answer 2

Instead of looping through the predictions why don't you import mean_squared_error from scikit learn's metrics module? I'm guessing you are optimising using the mean squared log error objective.

Firstly, Log transform the Actual and predicted values and find the mean squared error

Example: from sklearn.metrics import mean_squared_error

mean_sq_error = mean_squared_error(np.log1p(yval), np.log1p(pred))

np.log1p(x) does the same as np.log(x+1)