2
$\begingroup$

Thought I had solved the problem but I'm having inconsistent issues with it so reaching out here.

I have a multilabel classification problem with four labels ['--','-','+','++'] and with a basic random forest model, I have significant performance issues with one label '-', while the other three labels are performing pretty decently.

model = RandomForestClassifier(random_state=42)
model_name = 'RFC_base'

grid_params={}

pipe = GridSearchCV(make_pipeline(model),scoring='recall_weighted',cv=5,param_grid=grid_params,n_jobs=-1)

          recall avg    recall (++) recall (+)  recall (-)  recall (--) 
RFC_base    0.848485    0.840909    0.871795    0.771429    0.893617

I'm well aware that GridSearchCV is over engineering, but I like to keep code consistent between tinkers while optimising.

I wanted to improve the recall score for '-', so created a custom scoring function that I thought would be maximising the recall score for '-'.

recall_neg_scorer = make_scorer(recall_score,average=None,labels=['-'],greater_is_better=True)

I've been tinkering with this to optimise it, hence trying to define the 'greater_is_better' param. So I do an actual GridSearchCV now changing some of the params.

model = RandomForestClassifier(random_state=42)
scaler = StandardScaler()

grid_params = {
    'randomforestclassifier__n_estimators': [81], #np.arange(start=60,stop=90,step=1),
    'randomforestclassifier__max_depth': np.arange(1,10,step=1),
    'randomforestclassifier__criterion': ['gini'], #,'entropy'],
    'randomforestclassifier__max_features': [7], #np.arange(1,X.shape[1]+1,step=1),
    'standardscaler': [StandardScaler(),MinMaxScaler(),None]
}

model_name = 'RFC_gscv_recall_neg'

pipe = GridSearchCV(make_pipeline(scaler,model),scoring=recall_neg_scorer,param_grid=grid_params,cv=5,n_jobs=-1)

What I'm finding is my scoring function doesn't actually optimise '-' when 'greater_is_better = True'

For instance, when GSCV runs on 'max_depth' it chooses 9 which gives :

          recall avg    recall (++) recall (+)  recall (-)  recall (--) 
RFC_gscv_9  0.836364    0.818182    0.871795    0.742857    0.893617

Which is performing worse on '-' than even the base case. When I force max_depth = 8 it gives :

          recall avg    recall (++) recall (+)  recall (-)  recall (--) 
RFC_force_8 0.842424    0.818182    0.871795    0.8        0.87234  

When I choose 'greater_is_better = False' it actively tries to minimise the score. Is there something I'm screwing up here or is there a known issue that I'm missing out?

Also I'm a bit new to stack so let me know if there's something I'm missing.

$\endgroup$

2 Answers 2

1
$\begingroup$

I would highly suggest doing some deeper Analysis on why the ("-") class is beeing predicted. Meaning what do you predict for class ("-") when no the correct class. Analysing this you could see that you are ordering class ("-") as class "++". Than you can look at the Input data and feature engineer Features that discriminate between These two class. Forcing you model to do this indirectly is a fair try, but model cant learn whats not to be found in the data. With feature Engineering you can enhance the data so that discrimination is better.

$\endgroup$
1
  • $\begingroup$ Thanks! So I've already gotten a confusion matrix plotted for this to be able to see which ones are being misclassified. I can see myself masking the true classifications for '-' and false classifications (potentially '++' or others) and then using describe to visualise the difference in the data between them. Why are they so similar and what else is being missed... For instance, if one factor is making them so similar, but doesn't show much affect between other classes, maybe I could remove. I hadn't thought about it this way. Thanks for your feedback! $\endgroup$ Nov 1, 2020 at 18:54
0
$\begingroup$

So I've just figured this out and I feel like a bit of an idiot. I didn't post this in the question - which is part of why I need to be better at stack.

So all of the scoring data was based on "test" scores. The optimised scoring data for the GSCV is obviously based on just the TRAINING data.

So I guess I was making some data leakage for my model when optimising based on test score data, and not the train score data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.