# Train/Test/Validation Set Splitting in Sklearn

How could I split randomly a data matrix and the corresponding label vector into a X_train, X_test, X_val, y_train, y_test, y_val with Sklearn? As far as I know, sklearn.cross_validation.train_test_split is only capable of splitting into two, not in three...

You could just use sklearn.model_selection.train_test_split twice. First to split to train, test and then split train again into validation and train. Something like this:

 X_train, X_test, y_train, y_test
= train_test_split(X, y, test_size=0.2, random_state=1)

X_train, X_val, y_train, y_val
= train_test_split(X_train, y_train, test_size=0.2, random_state=1)

• Yes, this works of course but I hoped for something more elegant ;) Never mind, I accept this answer. – Hendrik Nov 17 '16 at 8:10
• I wanted to add that if you want to use the validation set to search for the best hyper-parameters you can do the following after the split: gist.github.com/albertotb/1bad123363b186267e3aeaa26610b54b – skd Jun 6 '18 at 16:34
• So what is the final train, test, validation proportion in this example? Because on the second train_test_split , you are doing this over the previous 80/20 split. So your val is 20% of 80%. The split proportions aren't very straightforward in this way. – Monica Heddneck Jun 14 '18 at 19:22
• I agree with @Monica Heddneck that the 64% train, 16% validation and 20% test splt could be clearer. It's an annoying inference you have to make with this solution. – Perry Jun 25 at 8:00

There is a great answer to this question over on SO that uses numpy and pandas.

The command (see the answer for the discussion):

train, validate, test = np.split(df.sample(frac=1), [int(.6*len(df)), int(.8*len(df))])


produces a 60%, 20%, 20% split for training, validation and test sets.

• I can see the .6 meaning 60%... but what does the .8 mean? – Tom Hale May 11 at 5:02
• @TomHale np.split will split at 60% of the length of the shuffled array, then 80% of length (which is an additional 20% of data), thus leaving a remaining 20% of the data. This is due to the definition of the function. You can test/play with: x = np.arange(10.0), followed by np.split(x, [ int(len(x)*0.6), int(len(x)*0.8)]) – 0_0 May 14 at 13:35

Most often you will find yourself not splitting it once but in a first step you will split your data in a training and test set. Subsequently you will perform a parameter search incorporating more complex splittings like cross-validation with a 'split k-fold' or 'leave-one-out(LOO)' algorithm.

Best answer above does not mention that by separating two times using train_test_split not changing partition sizes wont give initially intended partition:

x_train, x_remain = train_test_split(x, test_size=(val_size + test_size))


Then the portion of validation and test sets in the x_remain change and could be counted as

new_test_size = np.around(test_size / (val_size + test_size), 2)
# To preserve (new_test_size + new_val_size) = 1.0
new_val_size = 1.0 - new_test_size

x_val, x_test = train_test_split(x_remain, test_size=new_test_size)


In this occasion all initial partitions are saved.

You can use train_test_split twice. I think this is most straightforward.

X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=1)
X_train, X_val, y_train, y_val = train_test_split(
X_train, y_train, test_size=0.25, random_state=1)


In this way, train, val, test set will be 60%, 20%, 20% of the dataset respectively.

Here's another approach (assumes equal three-way split):

# randomly shuffle the dataframe
df = df.reindex(np.random.permutation(df.index))

# how many records is one-third of the entire dataframe
third = int(len(df) / 3)

# Training set (the top third from the entire dataframe)
train = df[:third]

# Testing set (top half of the remainder two third of the dataframe)
test = df[third:][:third]

# Validation set (bottom one third)
valid = df[-third:]


This can be made more concise but I kept it verbose for explanation purposes.

Given train_frac=0.8, this function creates a 80% / 10% / 10% split:

import sklearn

def data_split(examples, labels, train_frac, random_state=None):
''' https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html
param data:       Data to be split
param train_frac: Ratio of train set to whole dataset

Randomly split dataset, based on these ratios:
'train': train_frac
'valid': (1-train_frac) / 2
'test':  (1-train_frac) / 2

Eg: passing train_frac=0.8 gives a 80% / 10% / 10% split
'''

assert train_frac >= 0 and train_frac <= 1, "Invalid training set fraction"

X_train, X_tmp, Y_train, Y_tmp = sklearn.model_selection.train_test_split(
examples, labels, train_size=train_frac, random_state=random_state)

X_val, X_test, Y_val, Y_test   = sklearn.model_selection.train_test_split(
X_tmp, Y_tmp, train_size=0.5, random_state=random_state)

return X_train, X_val, X_test,  Y_train, Y_val, Y_test


Adding to @hh32's answer, while respecting any predefined proportions such as (75, 15, 10):

train_ratio = 0.75
validation_ratio = 0.15
test_ratio = 0.10

# train is now 75% of the entire data set
# the _junk suffix means that we drop that variable completely
x_train, x_test, y_train, y_test = train_test_split(dataX, dataY, test_size=1 - train_ratio)

# test is now 10% of the initial data set
# validation is now 15% of the initial data set
x_val, x_test, y_val, y_test = train_test_split(x_test, y_test, test_size=test_ratio/(test_ratio + validation_ratio))

print(x_train, x_val, x_test)


Extension of @hh32's answer with preserved ratios.

# Defines ratios, w.r.t. whole dataset.
ratio_train = 0.8
ratio_val = 0.1
ratio_test = 0.1

# Produces test split.
x_remaining, x_test, y_remaining, y_test = train_test_split(
x, y, test_size=test_ratio)

# Adjusts val ratio, w.r.t. remaining dataset.
ratio_remaining = 1 - ratio_test
`
$$R_{remaining} \cdot R_{new} = R_{old}$$