0
$\begingroup$

I want to do the following:

  • train a model using cross-validation
  • use the model for prediction (test dataset)
  • check the algorithmic bias towards some features values

I wonder if what I am doing is right? or there is another way?. Also, I have some feature that has many values. Is there a better way to split test data into subsets based on the values (like below split into male and female)?

from sklearn.model_selection import StratifiedKFold
from sklearn.ensemble import RandomForestClassifier as RFC
from sklearn.metrics import f1_score
include_fesatures=all features except demographics like gender and region
X=full_DF[include_fesatures]# include_feature are numeric here
y=X.pop('target')

X, X_test, y, y_test = train_test_split(X, y, test_size=0.2,random_state=1,stratify=y)
skf = StratifiedKFold(n_splits=10)
clf = RFC()   
c=0
for train_index, test_index in skf.split(X,y):
    Xtr,Xte,ytr,yte=X.iloc[train_index],X.iloc[test_index],y.iloc[train_index],y.iloc[test_index]
    clf.fit(Xtr,ytr)
    y_predict = clf.predict(Xte)
    acc = f1_score(yte, y_predict)#accuracy_score(yte, y_predict)
    c= c+ acc
print ('Accuracy:', float(c)/10)

indexes=X_test.index
mixed_df=full_DF[full_DF.index.isin(indexes)]
mdf_idx=mixed_df[mixed_df['gender']=='M'].index
fdf_idx=mixed_df[mixed_df['gender']=='F'].index
mX,my=X_test[X_test.index.isin(mdf_idx)],y_test[y_test.index.isin(mdf_idx)]
fX,fy=X_test[X_test.index.isin(fdf_idx)],y_test[y_test.index.isin(fdf_idx)]

print(f1_score(y_test, clf.predict(X_test)))
print(f1_score(my, clf.predict(mX)))
print(f1_score(fy, clf.predict(fX)))
$\endgroup$

1 Answer 1

0
$\begingroup$

Cross validation generally is used to assess model performance. Usually, you will train the model on some part of the data (e.g. 4/5 in 5-fold CV) and test on the remaining part (1/5). This process is repeated for all folds. Also see Introduction to Statistical Learning (Chapter 5.1) for a good overview.

What you miss in your approach is hyperparameter tuning. This is important for many models. So far you only train with standard parameter. One way of doing this (just an example, see sklearn docs for details) could be:

  • Do test/train split
  • Tune using 5-fold cv on train data
  • Fit proper model with "best" parameter on train data
  • Test overall performance on test data

Minimal example:

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.experimental import enable_halving_search_cv  # noqa
from sklearn.model_selection import HalvingGridSearchCV
from sklearn.metrics import accuracy_score

X, y = datasets.load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0, stratify=y)

param_grid = {'max_depth': [10,50,100],
              'min_samples_split': [2, 5, 10, 20]}
base_estimator = RandomForestClassifier(random_state=0)

sh = HalvingGridSearchCV(base_estimator, param_grid, cv=5,
                         factor=2, resource='n_estimators',
                         max_resources=30).fit(X_train, y_train)

print(sh.best_estimator_)
print("CV Score",sh.best_score_)

clf = RandomForestClassifier(max_depth=50, min_samples_split=20, n_estimators=24, random_state=0).fit(X_train, y_train)
y_pred = clf.predict(X_test)
print("Predicted Acc." , round(accuracy_score(y_test, y_pred),2))
$\endgroup$
2
  • $\begingroup$ Dear Dr.Peter. thank you for the reply. Actually, my aim is to compare between multiple models or classifiers. I do not target hyperparameter tuning, just want to use the models with their default parameters in sklearn. In a paper, they mentioned they used a cross-validated model in checking model performance for groups like male, female in a test dataset. Since cross_val_score does not return a model to use for predictions test data later, I though of this way. what do you think? Is it right?. $\endgroup$
    – Saif
    Commented Feb 18, 2022 at 14:44
  • $\begingroup$ Training a model on 4/5 of the data and prediction on 1/5 for each fold (to get a score) seems a reasonable approach to compare models $\endgroup$
    – Peter
    Commented Feb 18, 2022 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.