My problem context:

  1. dataset too big to fit into memory.
  2. binary classification [0,1]
  3. 30 csv files in a directory with exactly 30,000 samples (rows)
  4. each file contains 15,000 0 class and 15,000 1 class (no unbalance)
  5. model is xgboost.XGBClassifier()

If I iterate through these 30 files, fit a model, and pickle the model to disk; is it possible to aggregate these 30 fitted models into a single model? I am aware of sklearn.ensemble.VotingClassifier but that is used for:


Soft Voting/Majority Rule classifier for unfitted estimators.

  • $\begingroup$ I am using XGBClassifier() for xgboost. It outperforms SGD by a significant margin and unfortunately doesn't have a warm_start-like feature because of the nature of decision trees. I was able to use xgboost xgb_model parameter as in xgboost.train(parameters, dtrain, num_boost_round=250, xgb_model='model.pkl') but its performance differs from the sklearn wrapper despite me setting the equivalent hyperparameters. $\endgroup$
    – Jarad
    Oct 6 '17 at 17:37
  • 1
    $\begingroup$ I think we need more context. Is this the same problem, split into 30 pieces, or 30 separate problems? $\endgroup$
    – CalZ
    Oct 7 '17 at 13:37
  • $\begingroup$ Some sklearn models allow a prefit option; there's been some chat about adding that to ensembles like VotingClassifier (github.com/scikit-learn/scikit-learn/issues/12297) and StackingClassifier (github.com/scikit-learn/scikit-learn/pull/16748). $\endgroup$ Mar 26 '20 at 12:55

Despite the downvote, the question is clear, and a common one I'm sure most stumble across after doing machine learning work for some time.

The goal was to make a stronger predictive model from multiple trained models.

Quote from my question:

is it possible to aggregate these 30 fitted models into a single model?

Answer: yes but there's no good functionality that allows you to do this in sklearn.

Verbose answer:

Imagine you have 30 CSV files that contain 15,000 0 class and 15,000 1 class samples. In other words, an equally balanced number of binary responses (no class imbalance). I generated these files myself because 1) the size of the data I'm working with is too big to fit into memory (point #1 of my original question) and 2) contains 97% class 0 and 3% class 1 (large class imbalance). My goal is to see if it's even possible to distinguish between a 0 or a 1 if the class imbalance issue was removed from the equation. If there were distinguishing features found, I'd want to know what those features were.

To generate each batch, I grabbed 15,000 1 responses, and randomly sampled from the 97% 15,000 more samples, joined into one dataset (30,000 samples total), then shuffled them at random.

batch files

I then went through each batch and trained an XGBClassifier() using optimal parameters found from GridSearchCV() for each. I then saved the model to disk (using Python's pickle capabilities).

At this point, you have 30 saved models.

/models/ directory looks like this:

['model_0.pkl', 'model_1.pkl', 'model_10.pkl', 'model_11.pkl', 'model_12.pkl',
 'model_13.pkl', 'model_14.pkl', 'model_15.pkl', 'model_16.pkl', 'model_17.pkl',
 'model_18.pkl', 'model_19.pkl', 'model_2.pkl', 'model_20.pkl', 'model_21.pkl',
 'model_22.pkl', 'model_23.pkl', 'model_24.pkl', 'model_25.pkl', 'model_26.pkl',
 'model_27.pkl', 'model_28.pkl', 'model_29.pkl', 'model_3.pkl', 'model_4.pkl',
 'model_5.pkl', 'model_6.pkl', 'model_7.pkl', 'model_8.pkl', 'model_9.pkl']

Again, going back to the original question - is it possible to aggregate these into a single model? I wrote a pretty basic python function for this.

def xgb_predictions(X):
  ''' returns predictions from 30 saved models '''

  predictions = {}
  for pkl_file in os.listdir('./models/'):
    file_num = int(re.search(r'\d+', pkl_file).group())
    xgb = pickle.load(open(os.path.join('models', pkl_file), mode='rb'))
    y_pred = xgb.predict(X)
    predictions[file_num] = y_pred

  new_df = pd.DataFrame(predictions)
  new_df = new_df[sorted(new_df.columns)]

  return new_df

It iterates through each file in a directory of saved models and loads them one at a time casting its own "vote" so-to-speak. The end result is a new dataframe of predictions.

Since this new dataset is smaller (X.shape x 30), it will fit into memory. I iterate through each batch file calling xgb_predictions(X), getting a dataframe of predictions, add the y column to the dataframe, and append it to a new file. I then read the full dataset of predictions and create a "level 2" model instance where X is the prediction data and y is still y.

So to recap, the concept is, for binary classification, create equally balanced class datasets, train a model on each, run through each dataset and let each trained model cast a prediction. This collection of predictions is, in a way, a transformed version of your original dataset. You build another model to try and predict y given the predictions dataset. Train it all in memory so you end up with a single model. Any time you want to predict, you take your dataset, transform it into predictions, then use your level 2 trained model to cast the final prediction.

This technique performed much better than any other single model did or even stacking with UNTRAINED prior models.

  • $\begingroup$ This should be selected as the answer. $\endgroup$ Jun 15 '18 at 23:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.