XGBoost speed issues

Question

I'm trying to optimize the hyperparameters for XGBoost, thus needing to run it multiple times with different parameters. However the time needed to run single XGBoost with the parameters provided below in the code takes an enormous time (since multiple iterations with big sample-sizes are needed for the optimization).

My colleague claims that for the sample-size of 20 and with 13 iterations (so 260 times to run XGBoost) it took 2h47' (167') (on 2 cores). However so far I'm incapable of seeing this kind of speeds. For me the provided code takes ~4' to run on all the machines that I have tried it on (always 2 core machines and using 2 threads). Thus in total for 260 XGBoost trainings it would take me 1040 minutes (~6 times more time).

Since the average num_boost_round used by the colleague's code is ~250, then I would expect on average, that a single XGBoost training would take even more than 4 minutes.

Since it was months ago when the fast version of the XGBoost was run, the configuration of the computer might be changed, since I was not able to reproduce this speed. Does anyone have a clue what might be the problem? How long does it take for you with your xgboost setup to run this code snippet?

The mnist-numbers dataset is available to download here: http://yann.lecun.com/exdb/mnist/

I'm using the XGBoost python package that is installed using pip: pip install xgboost.

The parameters used are included in the code-snippet

The code:

import xgboost as xgb
import time
import os
import struct
import numpy as np

# Read MNIST files
def read_images(images_name):
    f = open(images_name, "rb")
    ds_images = []
    mw_32bit = f.read(4)
    n_numbers_32bit = f.read(4)
    n_rows_32bit = f.read(4)
    n_columns_32bit = f.read(4)
    mw = struct.unpack('>i', mw_32bit)[0]
    n_numbers = struct.unpack('>i', n_numbers_32bit)[0]
    n_rows = struct.unpack('>i', n_rows_32bit)[0]
    n_columns = struct.unpack('>i', n_columns_32bit)[0]
    try:
        for i in range(n_numbers):
            image = []
            for r in range(n_rows):
                for l in range(n_columns):
                    byte = f.read(1)
                    pixel = struct.unpack('B', byte)[0]
                    image.append(pixel)
            ds_images.append(image)
    finally:
        f.close()
    return ds_images


def read_labels(labels_name):
    f = open(labels_name, "rb")
    ds_labels = []
    mw_32bit = f.read(4)
    n_numbers_32bit = f.read(4)
    mw = struct.unpack('>i', mw_32bit)[0]
    n_numbers = struct.unpack('>i', n_numbers_32bit)[0]
    try:
        for i in range(n_numbers):
            byte = f.read(1)
            label = struct.unpack('B', byte)[0]
            ds_labels.append(label)
    finally:
        f.close()
    return ds_labels


def read_dataset(images_name, labels_name):
    images = read_images(images_name)
    labels = read_labels(labels_name)
    assert len(images) == len(labels)
    return (images, labels)


def create_datasets(sample_dir):
    image_file = os.path.join(sample_dir, 'train-images-idx3-ubyte')
    label_file = os.path.join(sample_dir, 'train-labels-idx1-ubyte')
    training_images, training_labels = read_dataset(image_file, label_file)
    image_file = os.path.join(sample_dir, 't10k-images-idx3-ubyte')
    label_file = os.path.join(sample_dir, 't10k-labels-idx1-ubyte')
    testing_images, testing_labels = read_dataset(image_file, label_file)
    dtrain = xgb.DMatrix(
        np.asmatrix(training_images),
        label=training_labels,
        nthread=2
    )
    dtest = xgb.DMatrix(
        np.asmatrix(testing_images),
        label=testing_labels,
        nthread=2
    )
    data_dict = {
        'dtrain': dtrain,
        'dtest': dtest,
        'training_labels': training_labels,
        'testing_labels': testing_labels
    }
    return data_dict


def main():
    data_dict = create_datasets("/path/to/mnist/numbers/sample")
    parameters = {
        'learning_rate': 0.1,
        'max_depth': 3,
        'gamma': 1,
        'min_child_weight': 20,
        'subsample': 0.8,
        'colsample_bytree': 0.4,
        'verbosity': 1,
        'objective': 'multi:softprob',
        'num_class': 10,
        'nthread': 2,
        'seed': 1,
    }
    start = time.time()
    model = xgb.train(
        parameters,
        data_dict['dtrain'],
        num_boost_round=100
    )
    end = time.time()
    print("Model training time: ", end-start)
    start2 = time.time()
    pred_train = model.predict(data_dict['dtrain'])
    end2 = time.time()
    print("Predict time: ", end2-start2)


if __name__ == '__main__':
    main()

Answer 1

Double-check that you are getting thread-based speed-ups. Vary nthread and see if increasing cores reduces training time.

You are only using two threads, nthread=2. It would be better to use all possible cores by setting nthread=-1. Even if you have only two cores on the current machine, it might help.

Then move to a machine that has more cores.