I am working on a binary classification task using SVM. The dataset is quite large so I don't want to use k-fold CV for parameter tuning, but instead a simple train-validation-test split. I have done the following:

X_train, X_test, y_train, y_test = train_test_split(
    X, y, stratify = y, test_size=0.2, random_state=1)

X_train, X_val, y_train, y_val = train_test_split(
    X_train, y_train, stratify = y_train, test_size=0.25, random_state=1)

So I have a 60-20-20 training-validation-test split. Since my validation set is predefined I want to use Sklearn's predefinedsplit. So I need to get the indices of my training and validation samples and set the validation indices to 0 and training indices to -1 so I tried the bottom answer of this question:

split_index = [0 if x in X_val.index else -1 for x in X_train.index]

But this returns a list of only -1's. I am unsure where I am going wrong. All my X and y _train, _val, _test are dataframes so .index can be applied. I have printed out X_train.index and X_val.index and both return Int64Index arrays of different lengths.

I also tried using Hypopt as one of the answers in the above link mentioned but it seems to be broken at the moment. Any suggestions as to how I should proceed?


The result that all values are equal to -1 is to be expected, since you're checking if the indices from your training set occur in your validation set, which by definition is not the case since your train and validation sets should not have the same observations. If you want to add the 0/1 indicator to you original dataset you have to use X instead of X_train:

split_index = [0 if x in X_val.index else -1 for x in X.index]

This will set the index to 0 if it occurs in X_val and -1 for all other cases.

  • $\begingroup$ Ugh I'm so stupid... Thank you so much for clearing this up! $\endgroup$
    – BenBernke
    Mar 31 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.