I'm trying to optimize the hyperparameters for XGBoost, thus needing to run it multiple times with different parameters. However the time needed to run single XGBoost with the parameters provided below in the code takes an enormous time (since multiple iterations with big sample-sizes are needed for the optimization).

My colleague claims that for the sample-size of 20 and with 13 iterations (so 260 times to run XGBoost) it took 2h47' (167') (on 2 cores). However so far I'm incapable of seeing this kind of speeds. For me the provided code takes ~4' to run on all the machines that I have tried it on (always 2 core machines and using 2 threads). Thus in total for 260 XGBoost trainings it would take me 1040 minutes (~6 times more time).

Since the average num_boost_round used by the colleague's code is ~250, then I would expect on average, that a single XGBoost training would take even more than 4 minutes.

Since it was months ago when the fast version of the XGBoost was run, the configuration of the computer might be changed, since I was not able to reproduce this speed. Does anyone have a clue what might be the problem? How long does it take for you with your xgboost setup to run this code snippet?

The mnist-numbers dataset is available to download here: http://yann.lecun.com/exdb/mnist/

I'm using the XGBoost python package that is installed using pip: pip install xgboost.

The parameters used are included in the code-snippet

The code:

import xgboost as xgb
import time
import os
import struct
import numpy as np

# Read MNIST files
def read_images(images_name):
    f = open(images_name, "rb")
    ds_images = []
    mw_32bit = f.read(4)
    n_numbers_32bit = f.read(4)
    n_rows_32bit = f.read(4)
    n_columns_32bit = f.read(4)
    mw = struct.unpack('>i', mw_32bit)[0]
    n_numbers = struct.unpack('>i', n_numbers_32bit)[0]
    n_rows = struct.unpack('>i', n_rows_32bit)[0]
    n_columns = struct.unpack('>i', n_columns_32bit)[0]
        for i in range(n_numbers):
            image = []
            for r in range(n_rows):
                for l in range(n_columns):
                    byte = f.read(1)
                    pixel = struct.unpack('B', byte)[0]
    return ds_images

def read_labels(labels_name):
    f = open(labels_name, "rb")
    ds_labels = []
    mw_32bit = f.read(4)
    n_numbers_32bit = f.read(4)
    mw = struct.unpack('>i', mw_32bit)[0]
    n_numbers = struct.unpack('>i', n_numbers_32bit)[0]
        for i in range(n_numbers):
            byte = f.read(1)
            label = struct.unpack('B', byte)[0]
    return ds_labels

def read_dataset(images_name, labels_name):
    images = read_images(images_name)
    labels = read_labels(labels_name)
    assert len(images) == len(labels)
    return (images, labels)

def create_datasets(sample_dir):
    image_file = os.path.join(sample_dir, 'train-images-idx3-ubyte')
    label_file = os.path.join(sample_dir, 'train-labels-idx1-ubyte')
    training_images, training_labels = read_dataset(image_file, label_file)
    image_file = os.path.join(sample_dir, 't10k-images-idx3-ubyte')
    label_file = os.path.join(sample_dir, 't10k-labels-idx1-ubyte')
    testing_images, testing_labels = read_dataset(image_file, label_file)
    dtrain = xgb.DMatrix(
    dtest = xgb.DMatrix(
    data_dict = {
        'dtrain': dtrain,
        'dtest': dtest,
        'training_labels': training_labels,
        'testing_labels': testing_labels
    return data_dict

def main():
    data_dict = create_datasets("/path/to/mnist/numbers/sample")
    parameters = {
        'learning_rate': 0.1,
        'max_depth': 3,
        'gamma': 1,
        'min_child_weight': 20,
        'subsample': 0.8,
        'colsample_bytree': 0.4,
        'verbosity': 1,
        'objective': 'multi:softprob',
        'num_class': 10,
        'nthread': 2,
        'seed': 1,
    start = time.time()
    model = xgb.train(
    end = time.time()
    print("Model training time: ", end-start)
    start2 = time.time()
    pred_train = model.predict(data_dict['dtrain'])
    end2 = time.time()
    print("Predict time: ", end2-start2)

if __name__ == '__main__':

Double-check that you are getting thread-based speed-ups. Vary nthread and see if increasing cores reduces training time.

You are only using two threads, nthread=2. It would be better to use all possible cores by setting nthread=-1. Even if you have only two cores on the current machine, it might help.

Then move to a machine that has more cores.

  • $\begingroup$ Yes, varying nthread reduces training time. nthread=2 was set in order for all different machines to be on equal setting, so XGBoost wouln't use all the cores available. I started to wonder, if it was possible, that it is caused by somekind of caching mechanism? As in when one uses jupyter notebook and reruns parts of the code? Also, I have noted that having different versions of other packages actually influences the performance. $\endgroup$ – LauritsT Apr 16 '20 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.