how to determine the best strategy for my machine learning model? For instance, let's consider a scenario where I am working with linear regression and want to compare three different approaches. The first approach involves using all features as inputs, the second approach entails manually selecting the most correlated feature as input, and the third approach involves implementing Principal Component Analysis (PCA). Given these three approaches, is it appropriate to evaluate each one using k-fold cross-validation without retraining the model, and then compare the results of the cross-validation to determine the best approach without using test dataset ?


2 Answers 2


If you want to compare your different models it is essential to have appropriate evaluation techniques and to perform the same method on all models to make them comparable.

In your scenario, the approach you mentioned evaluating using the k-fold cross-validation is definitely appropriate. However, keep in mind that it would be even better to still have a separate test set (not part of the cross-validation) to have a final evaluation.

A slightly optimized approach would be:

  1. If possible, split your data set into a training, validation, and test set.

  2. Train your three different models (linear regression with all features, linear regression with correlated features and linear regression with PCA features).

  3. Evaluate the performance on the k-fold cross-validation of the validation set.

  4. Evaluate the performance on the new, unseen test set.

That way, you have two ways of comparing the approaches.

  • $\begingroup$ i can't split the data into a training, validation, and test set because i have a small data set . i divided the data into 20-80 ratio then i evaluated the models using kfold cross validation without doing the step 2 that was mentioned in your comment . after comparing the three models , i did retrain the selected model on the whole dataset and did a final evaluation using the test dataset . $\endgroup$ Jun 6 at 18:45
  • $\begingroup$ Ok, sorry for misunderstanding. Then your approach is reasonable. Then you could use the results from the k-fold cross validation for hyper parameter tuning since you do not have a validation set. $\endgroup$
    – technik
    Jun 7 at 19:32

Yes, it is appropriate to evaluate each approach using k-fold cross-validation without retraining the model and compare the results to determine the best approach.

In your scenario, you have three different approaches for linear regression:

1.Using all features as inputs: This is the baseline approach where you include all available features as inputs for your linear regression model.

2.Manual feature selection: In this approach, you manually select the most correlated feature and use it as the input for your linear regression model. This is a way to investigate if a specific feature has a strong relationship with the target variable.

3.Principal Component Analysis (PCA): PCA is a dimensionality reduction technique that transforms the original features into a set of orthogonal components. In this approach, you would use the principal components as inputs for your linear regression model, reducing the dimensionality of the feature space.

To compare these three approaches, you can perform k-fold cross-validation for each approach, where the data is divided into k folds, and the model is trained and evaluated k times using different train-test splits. This helps to estimate the performance of each approach and assess its generalization capabilities.

By evaluating each approach using cross-validation, you can obtain performance metrics such as mean squared error (MSE), mean absolute error (MAE), or R-squared, and compare these metrics across the different approaches. You can then select the approach that achieves the best performance on average across the cross-validation folds.

Remember to appropriately set the value of k in k-fold cross-validation based on the size of your dataset and the desired trade-off between computation time and estimation accuracy.

  • $\begingroup$ could you please explain more the last point of your comment , i worked with the recommended value of k which is 5 and i have a small dataset. $\endgroup$ Jun 7 at 17:09
  • $\begingroup$ Using a higher value of k (e.g., 10) may lead to more reliable estimates of model performance, but it can also increase the computational cost because the model needs to be trained and evaluated more times. Using a smaller value of k (e.g., 5) can reduce the computation time, but it may also introduce more variability in the estimated performance metrics because each fold has a smaller sample size for training and testing. $\endgroup$ Jun 8 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.