speed up the fit of model

What are the ways to speed up the fit of the model on large files (more than 200 mb)? I tried to divide into pieces (chunks) or use dask but the effect is unsatisfactory. I use 16GB RAM and i7 2.2GHz processor 4 core. It remains to buy a cloud service or use a better computer? My clasiffier

clf = xgb.XGBClassifier(n_estimators=500,
n_jobs=4,
max_depth=9,
learning_rate=0.05,
subsample=0.7,
colsample_bytree=0.7,
missing=-999,
gamma = 0.1)
# Number of variable > 400
# Number of rows > 200000

• what type of model? Jul 18 '19 at 19:00
• I tried with xboost. In general its binary classification problem. Jul 18 '19 at 19:02
• How many observations and variables do you have in your dataset? How many variables are categorical and how many levels they have? How do you encode categorical variables? Note that naive one-hot encoding might significantly increase the number of variables passed to xgboost. What parameters do you use (e.g. max depth, number of trees, sample rates and learning rate)? What is your current run times? P.S. You might want to include this information directly in your question. Jul 18 '19 at 20:48
• What about categorical variables? Can you include this information as well? Jul 19 '19 at 7:20
• less than 50 variables. Jul 19 '19 at 10:35

One thing you could try is to use a "normal" Logit, as it is computationally not very expensive. Using Lasso, Elastic Net, or Ridge can yield good results (often similar to boosting) shown here.

Here is a recent code example for Logit/Lasso.

If you would like to stick with boosting, you may check LightGBM as it tends to be faster. XGBoost tends to be "heavy" in terms of data handling. LightGBM (as the name indicates) aims at resolving this problem and I have had good experiences so far.

Here is the link to the the LightGBM docs.

• I will check LightGBM. What about techniques? I looking a more general solution. Jul 18 '19 at 19:57
• I guess splitting the file etc will not give you an advantage if you mean this by saying „techniques“. I think optimizing data handling is key. Also check Logit as a baseline. It should be fast. Jul 18 '19 at 20:07
• Oh... you could also go on Kaggle if it is only for one project. There you can run a Kernel on GPU for free. kaggle.com Jul 18 '19 at 20:12
• Fine, but if its for job purpose? Jul 18 '19 at 20:14
• Ähm... maybe not due to data protection issues. Jul 18 '19 at 20:34

You can try the following two techniques or even a combination of both

1. in case you are using one-hot encoding for categorical variables and have variables with many levels (i.e. categorical variables with high cardinality), you could first lump rare levels together and then encode or just use another approach to encode categorical variables (e.g. target encoding).

2. use only a small subset of observations (rows) to identify the most relevant variables (columns) and then refit the model using only these variables on the entire dataset. If the quality of the final model would be not satisfactory you’ll need to include more variables.