0
$\begingroup$

I got similar results in 2 models which consists of similar algorithms.

Model 1 with cv=10 has a f1'micro' of 0.941. See code below. Model 2 only train test split (no cv) has f1'micro' 0.953.

Now here is my understanding problem. Before I did a Grid-Search to find best hyperparameters. Now I would like to do just a cross validation to train the dataset. Like the red marked in the picture. In the code there is still the Grid Search inside.

Question 1: Is this code doing that what I want? (is that a cross validation to train the dataset?)

Question 2: when I have 2 models like in the picture, model 1 with cross validation (red marked) and model 2 with train validation test data -> what are the reasons to choose model 1 with cross validation and why?

enter image description here

X = df.drop('columnA', axis=1)
y = df.columnA

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.20, random_state=42)

xgb_params = {
    'max_depth': [6],
    'learning_rate': [0.3]}

xgb_clf = GridSearchCV(XGBClassifier(random_state=42),
                      xgb_params,
                      cv=10,
                      n_jobs=-1,
                      verbose=True)

xgb_clf.fit(X_train, y_train)


xgb_pred = xgb_clf.predict(X_test)
print(accuracy_score(xgb_pred, y_test))
print(f1_score(xgb_pred, y_test, average='micro'))

Im sorry if my point of view is strange, but I have a lack of knowledge and Im confused about Cross Validation and Kfold and how to use it.

$\endgroup$
2
  • $\begingroup$ k-fold CV doesn't produce a final model; are you using these two validation techniques to select hyperparameters? $\endgroup$ – Ben Reiniger Nov 1 '20 at 17:05
  • $\begingroup$ @BendReiniger you are right. I updatet the question. The cross validation is running in the code to find the best hyperparameter. $\endgroup$ – martin Nov 1 '20 at 18:25
2
$\begingroup$

I think you confused some technical names. Cross-Validation is the name of the procedure, and it has some techniques or approaches such as k-fold cross-validation, train test split, etc. All are techniques to measure the performance of a model.

In your case, you have the first model that is assessed using 10-fold cross-validation and has an f1-score of 0.941, and the second model is assessed using the train test split approach and has an f1-score of 0.953. In this case, choosing the better model depends on what you want to give the privilege. In other words, whether you focus more on False Predictions or True Predictions, or False Negatives of False Positives. For this purpose, check the confusion matrix below.

enter image description here

There are several scenarios which mainly are:

  1. Using Recall metric

You can use it if False Negatives are more important for you. In the medical analysis, for example, False Negatives are tried to be minimized and scientist measures model's performance essentially by using Recall. Because, for example, it is a more acceptable case to predict a person cancer but actually that person is healthy while compared to the case where a person is labeled as healthy but in actual that person has cancer.

  1. Using Precision metric

It is preferred when False Positives are more important than False Negatives.

  1. Using F1-score

It helps to identify the state of incorrectly classified samples. In other words, False Negative and False Positives are attached more importance.

  1. Using Accuracy score

It is mostly used when True Positive and True Negatives are prioritized.

So back to your question, you should not choose the model that best 'fits' (or performs) in a test, but you should choose a model that fits your demands most and the test must be chosen depending on the demands. If you want to have balanced False Negatives and False positives use the f-score to choose a model, which is the model2 in the above-mentioned case. However, if you focus on a different target, make your choice according to a performance metric.

$\endgroup$
3
  • $\begingroup$ Thank you very much. It´s a multiclass model and dataset is a bit imbalanced. That helps me a lot. $\endgroup$ – martin Oct 31 '20 at 20:41
  • $\begingroup$ Practically, when we say "cross validation" (CV) we mean k-fold CV, which is different from the train/test split procedure. $\endgroup$ – desertnaut Oct 31 '20 at 21:16
  • $\begingroup$ This doesn't seem to answer the question, since OP is using f1 score in both cases. $\endgroup$ – Ben Reiniger Nov 1 '20 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.