I am using chunks of 100000 rows at a time from the CSV file to train the a simple LASSO model.

How do I combine all of these models trained from these different chunks? I would like to use all these trained models for prediction.

I am familiar with DASK and other alternatives but I would like to use Pandas.

pipelines = {
    'lasso' : make_pipeline(StandardScaler(), Lasso(random_state=123))

for key, value in pipelines.items():
    print( key, type(value) )

# Lasso hyperparameters
lasso_hyperparameters = { 
    'lasso__alpha' : [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 10] 

hyperparameters = {
'lasso' : lasso_hyperparameters

# Create empty dictionary called fitted_models
fitted_models = {}

# Create cross-validation object from pipeline and hyperparameters
model = GridSearchCV(pipeline, hyperparameters[name], cv=10, n_jobs=-1)

def train(X_train, y_train):  
    # Fit model on X_train, y_train
    model.fit(X_train, y_train)

    # Store model in fitted_models[name] 
    fitted_models[name] = model

    # Print '{name} has been fitted'
    print(name, 'has been fitted.')
    print ("__________________________________")
    print (model.cv_results_)

for df in pd.read_csv('train_V2.csv', chunksize=100000):
    df = df.dropna()
    df = pd.get_dummies(df, columns=['matchType'])
    df_train = df.drop(['Id', 'groupId', 'matchId'], axis = 1)
    y = df_train.winPlacePerc       
    X = df_train.drop('winPlacePerc', axis=1)
    X_train, X_test, y_train, y_test = train_test_split(X, y, 
    X_train = np.asarray(X_train)
    X_test = np.asarray(X_test)
    y_train = np.asarray(y_train)
    y_test = np.asarray(y_test)

    train(X_train, y_train)
  • $\begingroup$ name should also be defined in your code. $\endgroup$
    – rnso
    Nov 20, 2018 at 1:21

3 Answers 3


What you are looking for is called "stochastic optimization". You don't need to fit separate models and then combine them.

  • $\begingroup$ Thanks. The reason I am doing this is because I have some 40 million rows and total data size is 650 mb. I started getting memory errors and hence decided to go with chunking $\endgroup$ Oct 21, 2018 at 2:44
  • $\begingroup$ Same thing - do one update per batch. $\endgroup$ Oct 21, 2018 at 5:21
  • $\begingroup$ Alright. Any reading material or source to read up? Thanks. $\endgroup$ Oct 22, 2018 at 16:48
  • $\begingroup$ en.wikipedia.org/wiki/Stochastic_gradient_descent For lasso you'll need something different, such as stochastic ADMM or friends. You don;t need to know the details, there are many open-source implementations. $\endgroup$ Oct 22, 2018 at 16:54

Consider using sklearn.linear_model.SGDRegressor with L1 penalty, which is equivalent to a Lasso.

This has a .partial_fit implementation to incrementally train the model with the chunked datasets, rather than training separate models.


for epoch in epochs:
    for df in pd.read_csv('train_V2.csv', chunksize=100000):

You can got 2 ways:

  • Either you ensemble all your models results with a Voting Regressor.
  • You incrementally train your model

Given your use case I would go for the incremental training. Nevertheless, not all scikit-learn models support incremental training. You can check the list of classifiers that support incremental training here.

As you are using a simple LASSO model, you can with SDGRegressor with L1 regularization.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.