I am using chunks of 100000 rows at a time from the CSV file to train the a simple LASSO model.

How do i combine all of these models trained from these different chunks? I would like to use all these trained models for prediction

I am familiar with DASK and other alternatives but i would like to use Pandas.

pipelines = {
    'lasso' : make_pipeline(StandardScaler(), Lasso(random_state=123))

for key, value in pipelines.items():
    print( key, type(value) )

# Lasso hyperparameters
lasso_hyperparameters = { 
    'lasso__alpha' : [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 10] 

hyperparameters = {
'lasso' : lasso_hyperparameters

# Create empty dictionary called fitted_models
fitted_models = {}

# Create cross-validation object from pipeline and hyperparameters
model = GridSearchCV(pipeline, hyperparameters[name], cv=10, n_jobs=-1)

def train(X_train, y_train):  
    # Fit model on X_train, y_train
    model.fit(X_train, y_train)

    # Store model in fitted_models[name] 
    fitted_models[name] = model

    # Print '{name} has been fitted'
    print(name, 'has been fitted.')
    print ("__________________________________")
    print (model.cv_results_)

for df in pd.read_csv('train_V2.csv', chunksize=100000):
    df = df.dropna()
    df = pd.get_dummies(df, columns=['matchType'])
    df_train = df.drop(['Id', 'groupId', 'matchId'], axis = 1)
    y = df_train.winPlacePerc       
    X = df_train.drop('winPlacePerc', axis=1)
    X_train, X_test, y_train, y_test = train_test_split(X, y, 
    X_train = np.asarray(X_train)
    X_test = np.asarray(X_test)
    y_train = np.asarray(y_train)
    y_test = np.asarray(y_test)

    train(X_train, y_train)
  • $\begingroup$ name should also be defined in your code. $\endgroup$ – rnso Nov 20 '18 at 1:21

What you are looking for is called "stochastic optimization". You don't need to fit separate models and then combine them.

  • $\begingroup$ Thanks. The reason I am doing this is because I have some 40 million rows and total data size is 650 mb. I started getting memory errors and hence decided to go with chunking $\endgroup$ – dinesh patro Oct 21 '18 at 2:44
  • $\begingroup$ Same thing - do one update per batch. $\endgroup$ – anymous.asker Oct 21 '18 at 5:21
  • $\begingroup$ Alright. Any reading material or source to read up? Thanks. $\endgroup$ – dinesh patro Oct 22 '18 at 16:48
  • $\begingroup$ en.wikipedia.org/wiki/Stochastic_gradient_descent For lasso you'll need something different, such as stochastic ADMM or friends. You don;t need to know the details, there are many open-source implementations. $\endgroup$ – anymous.asker Oct 22 '18 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.