# XGBoost speed issues

I'm trying to optimize the hyperparameters for XGBoost, thus needing to run it multiple times with different parameters. However the time needed to run single XGBoost with the parameters provided below in the code takes an enormous time (since multiple iterations with big sample-sizes are needed for the optimization).

My colleague claims that for the sample-size of 20 and with 13 iterations (so 260 times to run XGBoost) it took 2h47' (167') (on 2 cores). However so far I'm incapable of seeing this kind of speeds. For me the provided code takes ~4' to run on all the machines that I have tried it on (always 2 core machines and using 2 threads). Thus in total for 260 XGBoost trainings it would take me 1040 minutes (~6 times more time).

Since the average num_boost_round used by the colleague's code is ~250, then I would expect on average, that a single XGBoost training would take even more than 4 minutes.

Since it was months ago when the fast version of the XGBoost was run, the configuration of the computer might be changed, since I was not able to reproduce this speed. Does anyone have a clue what might be the problem? How long does it take for you with your xgboost setup to run this code snippet?

I'm using the XGBoost python package that is installed using pip: pip install xgboost.

The parameters used are included in the code-snippet

The code:

import xgboost as xgb
import time
import os
import struct
import numpy as np

f = open(images_name, "rb")
ds_images = []
mw = struct.unpack('>i', mw_32bit)[0]
n_numbers = struct.unpack('>i', n_numbers_32bit)[0]
n_rows = struct.unpack('>i', n_rows_32bit)[0]
n_columns = struct.unpack('>i', n_columns_32bit)[0]
try:
for i in range(n_numbers):
image = []
for r in range(n_rows):
for l in range(n_columns):
pixel = struct.unpack('B', byte)[0]
image.append(pixel)
ds_images.append(image)
finally:
f.close()
return ds_images

f = open(labels_name, "rb")
ds_labels = []
mw = struct.unpack('>i', mw_32bit)[0]
n_numbers = struct.unpack('>i', n_numbers_32bit)[0]
try:
for i in range(n_numbers):
label = struct.unpack('B', byte)[0]
ds_labels.append(label)
finally:
f.close()
return ds_labels

assert len(images) == len(labels)
return (images, labels)

def create_datasets(sample_dir):
image_file = os.path.join(sample_dir, 'train-images-idx3-ubyte')
label_file = os.path.join(sample_dir, 'train-labels-idx1-ubyte')
image_file = os.path.join(sample_dir, 't10k-images-idx3-ubyte')
label_file = os.path.join(sample_dir, 't10k-labels-idx1-ubyte')
dtrain = xgb.DMatrix(
np.asmatrix(training_images),
label=training_labels,
)
dtest = xgb.DMatrix(
np.asmatrix(testing_images),
label=testing_labels,
)
data_dict = {
'dtrain': dtrain,
'dtest': dtest,
'training_labels': training_labels,
'testing_labels': testing_labels
}
return data_dict

def main():
data_dict = create_datasets("/path/to/mnist/numbers/sample")
parameters = {
'learning_rate': 0.1,
'max_depth': 3,
'gamma': 1,
'min_child_weight': 20,
'subsample': 0.8,
'colsample_bytree': 0.4,
'verbosity': 1,
'objective': 'multi:softprob',
'num_class': 10,
'seed': 1,
}
start = time.time()
model = xgb.train(
parameters,
data_dict['dtrain'],
num_boost_round=100
)
end = time.time()
print("Model training time: ", end-start)
start2 = time.time()
pred_train = model.predict(data_dict['dtrain'])
end2 = time.time()
print("Predict time: ", end2-start2)

if __name__ == '__main__':
main()


Double-check that you are getting thread-based speed-ups. Vary nthread and see if increasing cores reduces training time.
You are only using two threads, nthread=2. It would be better to use all possible cores by setting nthread=-1. Even if you have only two cores on the current machine, it might help.
• Yes, varying nthread reduces training time. nthread=2 was set in order for all different machines to be on equal setting, so XGBoost wouln't use all the cores available. I started to wonder, if it was possible, that it is caused by somekind of caching mechanism? As in when one uses jupyter notebook and reruns parts of the code? Also, I have noted that having different versions of other packages actually influences the performance. Apr 16, 2020 at 15:53