1
$\begingroup$

I would like to grid search pool classifiers hyper parameter of OLA() ( Overall Local Accuracy ) model from deslib python package.

from sklearn.datasets import make_classification


from sklearn.model_selection import RepeatedStratifiedKFold

from sklearn.model_selection import cross_val_score


from deslib.dcs.ola import OLA


from sklearn.model_selection import GridSearchCV

from sklearn.linear_model import LogisticRegression

from sklearn.tree import DecisionTreeClassifier

from sklearn.naive_bayes import GaussianNB

Then :

X , y = make_classification( n_samples = 10000 , n_features = 20 , n_informative = 15 , n_redundant = 5 , random_state = 999 )

model = OLA()

cv = RepeatedStratifiedKFold( n_splits = 10 , n_repeats = 3 , random_state = 999 )

grid = dict()

grid[ 'pool_classifiers' ] = [ [ LogisticRegression() , DecisionTreeClassifier() , GaussianNB() ] ,
                               [ LogisticRegression() , DecisionTreeClassifier() ] ]

search = GridSearchCV( model , grid , scoring = 'accuracy' , cv = cv )

search_results = search.fit( X , y )

But the following error message is raised :

NotFittedError: This LogisticRegression instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.

It means model in the pool must be fitted prior to grid searching but I thought fitting took place on each train fold of the cv step.

Does it mean I must fit myself each model on training folds ?

Thanks for helping on this topic.

$\endgroup$
1
$\begingroup$

Here is my solution to this problem :

Let's say I have the classifiers pools I want to cross validate :

grid = dict()

grid[ 'pool_classifiers' ] = [ ( 'pool_01' , [ LogisticRegression() , DecisionTreeClassifier() , GaussianNB() ] ) , 
                               ( 'pool_02' , [ LogisticRegression() , DecisionTreeClassifier() ] ) ]

Then I can use the following function :

def ola_cv( X , y , cv ) :

  scores = dict()

  for pool_classifiers in grid[ 'pool_classifiers' ] :

      scores[ pool_classifiers[ 0 ] ] = list()

  for train_ix , test_ix in cv.split( X , y ) :

    X_train = X[ train_ix , : ]
    y_train = y[ train_ix ]

    X_test = X[ test_ix , : ]
    y_test = y[ test_ix ]

    for pool_classifiers in grid[ 'pool_classifiers' ] : 

      for model in pool_classifiers[ 1 ] :

        model.fit( X_train, y_train )

      ola = OLA( pool_classifiers = pool_classifiers[ 1 ] )

      ola.fit( X_train , y_train )

      y_pred = ola.predict( X_test )

      score = accuracy_score( y_test , y_pred )

      scores[ pool_classifiers[ 0 ] ].append( score )

  for k in scores.keys():

    print( f'pool_classifiers : {k} | accuracy : {np.mean(scores[ k ])} ({np.std(scores[ k ])})')

  return scores

to obtain for each classifiers pool mean accuracy ( and standard deviation ) on 10 * 3 = 30 folds.

NB : Mayne grid should be an argument of the ola_cv function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.