How to split and train a model for data in biology

Question

I am using gene expression data that are float numbers and want to train classifiers in view of binary classification. Since I am a novice in this field I have some questions:

1. The first classifier I am using is SVM. I am using sklearn tools which require a split of the data set in training and test data sets. As far as I know, in order to build the model one needs a splitting of the data set in train and validation data sets (finding the parameters of the model), and for fine-tuning of hyperparameters, one will need a test data set. Interestingly, given what I found in going through sklearn documentation, there is suggested a split in train and test data set only. There is no assertion on the validation data set. So, I am doubtful, If I am running the classifier correctly.

Here is the code that I am using:

   from sklearn.model_selection import train_test_split
   from sklearn.utils import shuffle
   from sklearn import svm
   from sklearn.metrics import roc_curve, auc
   xall, yall = shuffle(x_sm, y_sm, random_state=21)
   x_train, x_test, y_train, y_test = train_test_split(xall, yall, test_size=0.3,   random_state=3, stratify=y_sm)
   x_train.shape, x_test.shape`
   classifier = svm.SVC(kernel='linear', gamma='auto', C=2,probability=True)
   classifier.fit(x_train, y_train)
   y_predict = classifier.predict(x_test)
   probs= classifier.predict_proba(x_test)
   fpr, tpr, threshold = roc_curve(y_test, preds)

Can somebody explain, if implicitly sklearn is using internally the triple split in training, validation and test data sets ? If not, how should I modify the code to include the known scheme of splitting in train, validation and test data sets ? If instead of random splitting, one uses k-fold cross-validation, will the result again be a splitting in two and not three data sets ?

2. Before training the model, I am using the standardization tools and PCA for feature and thus dimensionality reduction. After that, I am taking the first 10 PCA-components in training the model as described above. Is this the correct way one would suggest ? Apart from PCA, there are other dimensionality reduction tools. Should one use a few of them, train the model and decide at the end, based on the model performance, which of the dimensionality reduction tools to use for a particular classifier?

3. Along with SVM, I would like to use 3 more classifiers on the same data set and compare their performance. Given the nature of the data I have, which classifier should I choose?

I will highly appreciate your answers. Thanks.

Answer 1

That's a lot of questions you're asking there. I'll try to answer one question after another. Keep in mind this is only my point of view and some people may disagree with my answers.

1. Train, test and validation datasets

First, I'll try to explain how I see the 3-splits of a dataset : train, test and validation sets.

1. Train set is the one used to train the model, the loss function is computed with this set, and then the model improves via gradient descent. This is the set where the model will usually give the best performances.
2. Test set is the set used to test the model and compute the metric (in your example, the metric is accuracy). This set is used to find the good architecture and hyperparameters of your model.
3. Validation set is less common because it is only used at the very end of your work. When you have trained all your models, found the best hyperparameters and architecture, you finally just try your model on the validation dataset and see if it works properly or not. This set is only used to 'Validate' the fact that your model is working as expected. This is why we usually don't use consider it, and sklearn only advise you to use train and test sets.

To give you an idea about how to split the datasets, we usually go for a split with :

• 80-90% on training set
• 15-5% on testing set
• <5% on validation set

The test size you chose in your code (30%) seems quite high and I would reduce it to 15 or 10%. I would not use a validation dataset as if your model works on your train and test sets, it is very likely to work on any other set.

2.About dimensionality reduction

We usually do not use dimensionality reduction when training models because we usually want the model to have the best accuracy possible and computing time is not important in most cases(not always). The question you should ask yourself is 'Do i need my model to run quickly ?':

• If you need it to run quickly, then using PCA or any dimensionality reduction technique is smart and will help reduce computation time.
• If run time is not an issue, then do not bother using dimensionality reduction technique as they are likely to decrease your model performances.

I don't know much about all the algorithms used to decrease dimensions, so if I had to reduce dimensions, I would use PCA whatever my classifier is.

3.Classification algorithms

They are some other algorithms you can use on your data to classify them. Here are the ones I would try (from least favorite to favorite one) :

• K-Nearest Neighbours
• Perceptron
• SVM
• Random Forest
• Multilayer perceptron

Sorry for the long answer, I hope this helps.

Answer 2

if implicitely sklearn is using internally the triple split in training, validation and test data sets ? If not, how should I modify the code to include the known scheme of splitting in train, validation and test data sets ? If instead of random splitting, one uses k-fold cross-validation, will the result again be a splitting in two and not three data sets ?

If you know the ground truth or true labels for all your data, then a train and a valid is the same as a train and test.

If you have a separate "test" set for which you don't know the labels, then you make a valid set from your train (labelled) set to help tweak and tune your model. After your model is ready, you use it on the test set.

The valid set in sklearn is when you separate out a subset of the training data, with labels intact, to validate (see how good) your model. this lets you fine tune the algorithm. So you train your model based on 80% or 90% of your labelled data, then using the remaining 20% labelled data in the form of the validation set, you figure out if the model is doing a good job or not.

So, you never need to split into three.

K fold treats a random subset of the training data as a validaton and the rest as training, so you just need to specify the number of times it does this (fold).