Consider the following data:

   import pandas as pd
    wine = pd.read_csv(r'wine_data.csv', names = ["Cultivator", "Alchol", "Malic_Acid", "Ash", "Alcalinity_of_Ash", "Magnesium", "Total_phenols", "Falvanoids", "Nonflavanoid_phenols", "Proanthocyanins", "Color_intensity", "Hue", "OD280", "Proline"])
    X = wine.drop('Cultivator',axis=1) #input
    y = wine['Cultivator'] #output

y is what i am trying to predict and X is the input and i will be using some sort of mlp classifier. What I want to do is split this data into test, training and validation and then apply K-folds. I'm struggling to see how you do this..

I know that I can obtain validation, test and training by using the following:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.25, random_state=1)

But what I want to do now is apply k folds such that for each fold I have 3 sets: validation, testing , training rather than just 2 sets.

I know I can use the below for Kfolds:

kf = KFold(n_splits = 5, shuffle = True, random_state = 2)

After converting to a numpy array I can then do this:

for train_index, test_index in kf.split(X_np):
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X_np[train_index], X_np[test_index]
    y_train, y_test = y_np[train_index], y_np[test_index]

But how do I get a 'validation_index'. In general the question is how do I use k-folds when I have 3 sets as opposed to just 2?

Also when do I normalize the data; do I normalize when i've split into X_train, X_test as above... or do I do it before?

Any help appreciated.


I don't think there's a builtin way to do it, but the two methods you've mentioned combine pretty nicely to do the job:

kf = KFold(n_splits = 5, shuffle = True, random_state = 2)

for train_index, test_index in kf.split(X):
    X_tr_va, X_test = X.iloc[train_index], X.iloc[test_index]
    y_tr_va, y_test = y[train_index], y[test_index]
    X_train, X_val, y_train, y_val = train_test_split(X_tr_va, y_tr_va, test_size=0.25, random_state=1)
    print("TRAIN:", list(X_train.index), "VALIDATION:", list(X_val.index), "TEST:", test_index)

(I also opted to keep things as dataframes. You could even use "nested cross-validation," using another CV instead of the train_test_split inside the loop, depending on your needs and computational budget.)

For the question of normalizing data, you don't want to let information from the testing fold affect the training, so normalize within the loop, using only the training set;

  • $\begingroup$ thank you, so for each fold, based on the training data for that fold - i normalize it? Then do i apply the mean/s.d. calculated from that on teh validation AND test set? $\endgroup$ – Maths12 Feb 20 '19 at 16:18
  • $\begingroup$ Yep, that seems like the way to go. (I suppose, depending on how exactly you're using the validation set, you could get away with normalizing training+validation together.) $\endgroup$ – Ben Reiniger Feb 20 '19 at 16:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.