Flow of machine learning model including code

Question

I'm towards the completion of my first data science project that will go into my GitHub portfolio. I'll be happy for some clarification regarding the machine learning models section: I got a little confused with the steps: evaluation model, baseline model, cross-validation, fit-predict, when to use (X, y), and when to split the data with train_test_split and use (X_train, y_train).

Dataset from Kaggle - Stroke Prediction: https://www.kaggle.com/datasets/fedesoriano/stroke-prediction-dataset?datasetId=1120859&sortBy=voteCount&searchQuery=models

The dataset contains 5110 observations with 10 attributes and a target variable: 'stroke'. The dataset is unbalanced, with 5% positive for stroke.

I tried to follow different projects, however, because each one has its own way, I got lost with what is the correct way and what is optional.

This is what I have so far:

Baseline model:

def load_data ():
    df = pd.read_csv('healthcare-dataset-stroke-data.csv')
    df=df.drop('id', axis=1)
    categorical = [ 'hypertension', 'heart_disease', 'ever_married','work_type', 'Residence_type', 'smoking_status']
    numerical = ['avg_glucose_level', 'bmi','age']
    y= df['stroke']
    X = df.drop('stroke', axis=1)
    return X,y,categorical, numerical

def baseline_model(X, y, model):
    transformer = ColumnTransformer(transformers=[('imp',SimpleImputer(strategy='median'),numerical),('o',OneHotEncoder(),categorical)])
    pipeline = Pipeline(steps=[('t', transformer),('p',PowerTransformer(method='yeo-johnson')),('m', model)])    
    cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
    scores = cross_val_score(pipeline, X, y, scoring='roc_auc', cv=cv, n_jobs=-1)
    return scores

X,y,categorical, numerical= load_data()
model = DummyClassifier(strategy='constant', constant=1)
scores = baseline_model(X, y, model)
print('Mean roc_auc: %.3f (%.3f)' % (np.mean(scores), np.std(scores)))

Output:

Mean roc_auc: 0.500 (0.000)

Evaluation model:

def evaluate_model(X, y, model):
    cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=42)
    scores = cross_val_score(model, X, y, scoring='roc_auc', cv=cv, n_jobs=-1)
    return scores

The models are:

def get_models():
    models, names = list(), list()
    models.append(DecisionTreeClassifier(random_state=42))    
    names.append('DT')
    models.append(RandomForestClassifier(random_state=42))    
    names.append('RF')
    models.append(XGBClassifier(random_state=42, eval_metric='error'))    
    names.append('XGB')
    models.append(LogisticRegression(solver='liblinear'))    
    names.append('LR')
    models.append(LinearDiscriminantAnalysis())
    names.append('LDA')
    models.append(SVC(gamma='scale'))
    names.append('SVM')
    return models, names

First model:

X,y,categorical, numerical= load_data()
print(X.shape, y.shape)

models, names = get_models()
results = list()

for i in range(len(models)):
    transformer = ColumnTransformer(transformers=[('imp',SimpleImputer(strategy='median'),numerical),('o',OneHotEncoder(),categorical)])
    pipeline = Pipeline(steps=[('t', transformer),('p',PowerTransformer(method='yeo-johnson')),('m', models[i])])    
    scores = evaluate_model(X, y, pipeline)
    results.append(scores)
    print('>%s %.3f (%.3f)' % (names[i], np.mean(scores), np.std(scores)))

Output:

(5110, 10) (5110,)
>DT 0.555 (0.034)
>RF 0.781 (0.030)
>XGB 0.809 (0.026)
>LR 0.839 (0.029)
>LDA 0.833 (0.030)
>SVM 0.649 (0.064)

Second model with SMOTE:

for i in range(len(models)):
    transformer = ColumnTransformer(transformers=[('imp',SimpleImputer(strategy='median'),numerical),('o',OneHotEncoder(),categorical)])
    pipeline = Pipeline(steps=[('t', transformer),('p',PowerTransformer(method='yeo-johnson', standardize=True)),('over', SMOTE()), ('m', models[i])])    
    scores = evaluate_model(X, y, pipeline)
    results.append(scores)
    print('>%s %.3f (%.3f)' % (names[i], np.mean(scores), np.std(scores)))

Output:

(5110, 10) (5110,)
>DT 0.579 (0.036)
>RF 0.765 (0.027)
>XGB 0.778 (0.031)
>LR 0.837 (0.029)
>LDA 0.839 (0.030)
>SVM 0.766 (0.040)

Logistic Regression Hyperparameter Tuning:

transformer = ColumnTransformer(transformers=[('imp',SimpleImputer(strategy='median'),numerical),('o',OneHotEncoder(),categorical)])
pipeline = Pipeline(steps=[('t', transformer),('p',PowerTransformer(method='yeo-johnson', standardize=True)),('s',SMOTE()),('m', LogisticRegression())])

param_grid = {
    'm__penalty': ['l1', 'l2'],
    'm__C': [0.001, 0.01, 0.1, 1, 10, 100]
}

cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=42)
grid = GridSearchCV(pipeline, param_grid, scoring='roc_auc', cv=cv, n_jobs=-1)

grid.fit(X, y)

print("Best hyperparameters: ", grid.best_params_)
print("Best ROC AUC score: ", grid.best_score_)

Output:

Best hyperparameters:  {'m__C': 0.01, 'm__penalty': 'l2'}
Best ROC AUC score:  0.8371495917165929

My questions are:

First:

Is it possible to end a project like this? OR Do I need to split the data into train/test subsets and make a prediction on unseen data after training with the best parameters? (See below)

Second:

When I use fit/predict:

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8, random_state=42)

logreg_pipeline = Pipeline(steps=[('t', transformer),('p',PowerTransformer(method='yeo-johnson', standardize=True)),('over', SMOTE()), ('m', LogisticRegression(C=0.01,penalty='l2',random_state=42))])

logreg_pipeline.fit(X_train,y_train)

logreg_tuned_pred = logreg_pipeline.predict(X_test)

print(classification_report(y_test,logreg_tuned_pred))

print('Accuracy Score: ',accuracy_score(y_test,logreg_tuned_pred))
print('ROC AUC Score: ',roc_auc_score(y_test,logreg_tuned_pred))

Output:

              precision    recall  f1-score   support

           0       0.98      0.74      0.85       960
           1       0.17      0.82      0.28        62

    accuracy                           0.75      1022
   macro avg       0.58      0.78      0.57      1022
weighted avg       0.94      0.75      0.81      1022

Accuracy Score:  0.7475538160469667
ROC AUC Score:  0.7826444892473119

Is this right and a necessary step? What is the right way to read this result? Do I compare it to the roc_auc score from the cross-validation/baseline model that was executed above?

I'd be happy to clarify any misunderstanding so that this whole issue will finally be clear to me. Thank you for your time and feedback :)

Answer 1

The purpose of a machine learning model is to make predictions on real-world data that isn’t known at model training time. As such, it’s best practice to always do a train-test split at the very beginning of any project, and only use the training data for training the model. The test data should not be used at all until your model is fully trained. To add to this, when tuning the model’s hyperparameters there is an additional subset of the training data used for validation, which is not used for training but for evaluating performance during training. You create train-test-splits of your input data, run through all of your models, and use your aggregate cross-validation score to choose one or two models to concentrate on improving. Based on your results, it looks like logistic regression is getting the highest score, and is probably a good fit for this type of problem – predicting whether an instance of the data is a member of the target or not (“stroke” or “not stroke”).

Once this is done, you can tune your model’s hyperparameters (using GridSearch like you’re doing for example) to determine the best parameters for things like regularization (the “C” parameter). Then, and only then, when you have selected your model, tuned the hyperparameters, and trained on your training data only, then you evaluate performance on your test data.

For the evaluation, it’s good to understand the performance of your model and what that represents, that’s what your metrics at the end are for. Precision is percentage of true positives over true positives and false positives, and recall is true positives over true positives plus false negatives. F1 score is the harmonic mean of these two values, ROC is the performance of the model at different classification thresholds. If the purpose of the model is to predict strokes, do you want a higher precision which would mean you detect more potential strokes at the risk of higher false positives? Or a higher recall which would mean all the instances classified as high risk of stroke are more likely to be high risk of stroke but at the cost of potentially missing some? Hth,