k-fold cross-validation: model selection or variation in models when using k-fold cross validation

Question

I am new to machine learning, though I have a background in statistics. But I had a question about $k$-fold cross-validation. So I understand the basic idea that we divide the dataset into $k$ partitions and then train a model on $k-1$ partitions while testing on the $k$th partition that was left out. So we don't want to train the model over the entire dataset, but instead over just our $k-1$ partitions.

My question was how do we handle changes in the model parameters with each fold in the process. Another way of asking this question is how do we choose the model to test? So when you train on $k-1$ folds, in each iteration there will be a subtle change in the parameter estimates--namely the $\beta_{0...p}$s will change. So when we test the model against the testing partition, we are not testing the same model each time--because the parameter coefficients are different.

Given that the $\beta$s change for each fold, how do we choose which model to use? And once we choose a model, do we need to test only this model for the $k-fold$ cross-validation, for otherwise we are testing different models and then averaging their prediction error.

Any clarification would be appreciated.

Answer 1

There are two main (set of) things your model needs to learn from training: Parameters and Hyperparameters.

Parameters are those that can be inferred (learnt) from data. These are internal to the model and will be calculated in the training phase during. The hyperparameters however, cannot be learnt from data and need to be decided (make assumptions) before training. As you come from statistics, an example could be modelling a distribution from a set of samples. Parameters could be mean and std, and hyperparameter could be selecting a Normal distribution or a Beta distribution.

If you don't have hyperparameters (e.g. simple regression model), you only need to train the model and let the algorithm find the optimal weights (parameters) using the data. However, if you want to use let's say Support Vector Machines, then you need to provide some hyperparameters such as the cost, kernel, parameters of the kernel, and so on... before starting the training.

Cross Validation can be used to find the optimal hyperparameters. It is done by creating a grid of potential hyperparameter values and training each combination of them against the whole data, performing k trainings of k-1 portions (or folds) of data. For each specific combination of hyperparameters the model is then trained and evaluated k times, getting a number of predictions equal to the number of folds (k) for each hyperparameter combination. The combination that yields better results based on the specified metric, is considered the best option (final model).

Have a look at this nice blog for further clarification.

Cross validation procedure:

Answer 2

I think @TitoOrt gets at the bulk of the confusion here, you're trying to explain how to get back the training parameters. When it seems like the point the authors are making is that you can use X-Fold (for avoiding confusion with my example KNN) to find a hyperparameter. Let me better explain with K-Nearest Neighbor:

Say you were trying to find the best K to use with the Iris set, here are error rates for K {2, 3, 4, 5}:

from sklearn import cluster, datasets, mixture
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split

data = sklearn.datasets.load_iris()
X = data["data"]
Y = data["target"]

accuracy = []
for n in range(2, 6):
    # 70% holdout for training
    x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=0.7)

    model = KNeighborsClassifier(n_neighbors=n)
    model.fit(x_train, y_train)

    score = model.score(x_test, y_test)
    accuracy.append((n, score))

# accuracy, number of clusters and the accuracy of the model
# [(2, 0.96190476190476193),
#  (3, 0.92380952380952386),
#  (4, 0.97142857142857142),
#  (5, 0.94285714285714284)]

However this brings up a really easy criticism, how do I know we didn't just get lucky in picking the data to pass into the model when K was set at 4? (Thankfully for this example, that's probably the seeing as there is only suppose to be 3 classes). So in order to overcome this we use X-Fold to average out the bias that comes from sampling data, and thus better trust the accuracy that we get from picking K.

accuracy = []
for n in range(2, 6):
    k_accuracy = []

    # splitting and running this loop 10 times
    kf = KFold(n_splits=10)
    for train_index, test_index in kf.split(X):

        model = KNeighborsClassifier(n_neighbors=n)
        model.fit(X[train_index], Y[train_index])

        score = model.score(X[test_index], Y[test_index])
        k_accuracy.append(score)

    # get the averaging of each 10 kfold runs
    score = sum(k_accuracy) / 10
    accuracy.append((n, score))

# accuracy
# [(2, 0.93333333333333335),
#  (3, 0.94666666666666688),
#  (4, 0.92666666666666675),
#  (5, 0.93333333333333335)]

First of all we've eliminated the K=4 option, and the actual K=3 emerges as the winner. With more folds our accuracy can be better trusted as well.

Let's connect this back to your original question though. Each radius attached to each cluster will be different every time you run this algorithm because we're fitting to the data. So yes you are right that you will never be able to exactly replicate the radius (parameter) choices that were made to get a certain accuracy, but that is not the goal of cross-validation. The more important aspect of folding was to pick the best K. Once we know that we can train to our hearts content.

Answer 3

The primary use of CV is for tuning the model. You use the average CV value to see how your model might behave on new data. Once you’ve settled for the best hyper parameters, your retrain on the whole train data.

Then people noticed that you can also use the models trained on each fold, by averaging their predictions on the test set. That’s the second way of using CV.

The best approach depend son the data. Sometimes retraining on the full set is better than averaging fold predictions, sometimes the converse is true.