How to do k-folds in python whilst splitting into 3 sets?

Question

Consider the following data:

   import pandas as pd
    wine = pd.read_csv(r'wine_data.csv', names = ["Cultivator", "Alchol", "Malic_Acid", "Ash", "Alcalinity_of_Ash", "Magnesium", "Total_phenols", "Falvanoids", "Nonflavanoid_phenols", "Proanthocyanins", "Color_intensity", "Hue", "OD280", "Proline"])
    X = wine.drop('Cultivator',axis=1) #input
    y = wine['Cultivator'] #output

y is what i am trying to predict and X is the input and i will be using some sort of mlp classifier. What I want to do is split this data into test, training and validation and then apply K-folds. I'm struggling to see how you do this..

I know that I can obtain validation, test and training by using the following:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.25, random_state=1)

But what I want to do now is apply k folds such that for each fold I have 3 sets: validation, testing , training rather than just 2 sets.

I know I can use the below for Kfolds:

kf = KFold(n_splits = 5, shuffle = True, random_state = 2)
X_np=np.array(X)
y_np=np.array(y)

After converting to a numpy array I can then do this:

for train_index, test_index in kf.split(X_np):
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X_np[train_index], X_np[test_index]
    y_train, y_test = y_np[train_index], y_np[test_index]

But how do I get a 'validation_index'. In general the question is how do I use k-folds when I have 3 sets as opposed to just 2?

Also when do I normalize the data; do I normalize when i've split into X_train, X_test as above... or do I do it before?

Any help appreciated.

Answer 1

I don't think there's a builtin way to do it, but the two methods you've mentioned combine pretty nicely to do the job:

kf = KFold(n_splits = 5, shuffle = True, random_state = 2)

for train_index, test_index in kf.split(X):
    X_tr_va, X_test = X.iloc[train_index], X.iloc[test_index]
    y_tr_va, y_test = y[train_index], y[test_index]
    X_train, X_val, y_train, y_val = train_test_split(X_tr_va, y_tr_va, test_size=0.25, random_state=1)
    print("TRAIN:", list(X_train.index), "VALIDATION:", list(X_val.index), "TEST:", test_index)

(I also opted to keep things as dataframes. You could even use "nested cross-validation," using another CV instead of the train_test_split inside the loop, depending on your needs and computational budget.)

For the question of normalizing data, you don't want to let information from the testing fold affect the training, so normalize within the loop, using only the training set;
https://stats.stackexchange.com/questions/77350/perform-feature-normalization-before-or-within-model-validation