0
$\begingroup$

My objective is to experiment with various approaches for different algorithms, identify the best approach for each algorithm, and subsequently determine the best overall algorithm from among these top approaches.

To accomplish this, I employed k-fold cross-validation to evaluate each approach. After conducting the evaluations, I selected the approach that yielded the most optimal metric.

To simplify things, let's consider linear regression. I tried different approaches by changing techniques and steps. To assess their performance, I evaluated each approach using k-fold cross-validation. Let's say I found that approach 2 performed the best for linear regression. Without training the model with new data, I moved on to the next algorithm, which was ANN. Following a similar process, I evaluated different approaches for ANN using k-fold cross-validation. This time, approach 3 turned out to be the best. Finally, I compared approach 2 for linear regression with approach 3 for ANN and chose the superior approach. I then trained the model using the selected approach and model.

Am I proceeding in the correct direction ?

$\endgroup$

1 Answer 1

0
$\begingroup$

Evaluation Metrics. For regression problems metrics like MSE or RMSE (is less sensitive to extreme values) are good defaults. For classification instead you can evaluate against accuracy if classes are balanced, otherwise look at the AUC of the ROC or PR (precision-recall) curves. In addition, the f1-score is also quite common, but in some other cases you may care more about errors and so the confusion matrix would give you an overview of the kind of error your model(s) made.

Basically, you pick one metric e.g. RMSE (for regression) and AUROC (AUC of ROC for classification), compute that for all your models and rank them accordingly. These metrics can be also used for selecting the best NN across training epochs (indeed you need to compute that on a validation-set.)

Compare and select models. Since training one model (of one kind) gives you only a point estimate of its overall performance, which is an approximation, because training and test data are just limited. Moreover, there could be randomness in the model and/or training process that, at each run, may yield a different model with different performance.

Especially if you have not so many data, K-fold cross-validation allows you to estimate the bias and variance of your model quite easily. K-fold cross-validation allows you to estimate uncertainties related to the model and data. However, say your $k=10$ so would obtain $k$ models for each kind of them: you evaluate then on the metrics you care, and, basically, obtain a distribution of performance for each model class.

You should then aggregate the performance on your evaluation metric, obtaining average performance (e.g. by taking the mean) but also its standard deviation (i.e. variability in model predictions). For example, say model-1 achieves the best average but its std is quite large, while model-2 1% lower but the std is almost zero. So, what model do you choose?

When selecting the model you should consider both mean and std, or the overall distribution. To help yourself you can inspect a boxplot of the performance distribution of each class of models, such that you can visualize both average performance and the their associated variability.

In alternative is also possible to compute a $p$-value that provides you the probability that one class of models (e.g. SVM) is better than another (e.g. neural-nets).

$\endgroup$
5
  • $\begingroup$ So, if I understand correctly, you're discussing the method of selecting the model ? because my question is divided into two parts: firstly, selecting the optimal approach for each model using k-fold cross-validation, and secondly, choosing the best model overall. it seems that considering the average and standard deviation is crucial for assessing the performance of the model. Initially, I was thinking that metrics like RMSE and R2 would be sufficient to determine whether one model outperforms another. nd I apologize if my questions appeared lacking in understanding cus I'm a beginner. $\endgroup$ Commented Jun 1, 2023 at 10:59
  • $\begingroup$ @SalahAmani I've updated my answer. Hope is clearer now, thanks $\endgroup$ Commented Jun 1, 2023 at 17:27
  • $\begingroup$ yes thanks a lot . your comment is very helpful. However, I'm still uncertain about the proper way to use k-fold cross-validation for comparing models. Would it be appropriate to assess the models using k-fold cross-validation on the training dataset, and once the optimal model is selected, retrain that model and evaluate its performance on the test dataset? $\endgroup$ Commented Jun 1, 2023 at 17:51
  • $\begingroup$ Yes, you can use the train-set, next determine the best model and finally train it on all the training data. K-fold cross-val is also useful to determine if over- or under-fitting is happening. Indeed, if you have many data point or even a separate validation set you can avoid k-fold cross-val to save some computation time $\endgroup$ Commented Jun 1, 2023 at 17:57
  • $\begingroup$ thanks for ur time $\endgroup$ Commented Jun 1, 2023 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.