eta and learning_rate different in xgboost

Question

I am creating a classification model using xgboost in python. I am using different eta values to check its effect on the model. My code is-

for eta in np.arange(0.2, 0.51, 0.03):
    xgb_model = xgboost.XGBClassifier(objective = 'multi:softmax', num_class = 5, eta = eta)
    xgb_model.fit(x_train, y_train)
    xgb_out = xgb_model.predict(x_test)
    print("For eta %f, accuracy is %2.3f" %(eta,metrics.accuracy_score(y_test, xgb_out)*100))

I expected different accuracies for some eta values, but to my surprise I got same accuracy for each eta. When I printed the model, I got this-

>>> print(xgb_model)
XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
       colsample_bytree=1, eta=0.5, gamma=0, learning_rate=0.1,
       max_delta_step=0, max_depth=3, min_child_weight=1, missing=None,
       n_estimators=100, n_jobs=1, nthread=None, num_class=5,
       objective='multi:softprob', random_state=0, reg_alpha=0,
       reg_lambda=1, scale_pos_weight=1, seed=None, silent=True,
       subsample=1)

Here you can see eta = 0.5, but learning_rate = 0.1. While in xgboost docs, learning_rate is an alias for eta. So how is this possible that both have different values?

Answer 1

Seems like eta is just a placeholder and not yet implemented, while the default value is still learning_rate, based on the source code. Good catch.

We can see from source code in sklearn.py that there seems to exist a class called 'XGBModel' that inherits properties of BaseModel from sklearn's API.

Tracing this to compat.py, we see there's an import statement:

from sklearn.base import RegressorMixin, ClassifierMixin

These are sklearn's Mixin classes for classifiers/regressors. Basically, they're the shells from which any specific classifier/regressor is generated, to guarantee that the descendant class has a score() method.

Looking carefully at the init function, we see:

__init__($self, /, *args, **kwargs)

From Yasoob's PythonTips (See link for examples of *args/**kwargs):

*args and **kwargs allow you to pass a variable number of arguments to a function. What variable means here is that you do not know beforehand how many arguments can be passed to your function by the user so in this case you use these two keywords...

*args is used to send a non-keyworded variable length argument list to the function.

**kwargs allows you to pass keyworded variable length of arguments to a function. You should use **kwargs if you want to handle named arguments in a function.

Now for a fun experiment to prove this. Try the following in your own notebook/console:

>>> XGBClassifier(eta=5, potato=1234)
XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
   colsample_bytree=1, eta=5, gamma=0, learning_rate=0.1,
   max_delta_step=0, max_depth=3, min_child_weight=1, missing=None,
   n_estimators=100, n_jobs=1, nthread=None,
   objective='binary:logistic', potato=1234, random_state=0,
   reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,
   silent=True, subsample=1)

See that our argument was stored in the object:

`...objective='binary:logistic', potato=1234, random_state=0,`

We're pretty sure potato is not a hyperparameter right? :)