238 votes
Accepted

Train/Test/Validation Set Splitting in Sklearn

You could just use sklearn.model_selection.train_test_split twice. First to split to train, test and then split train again into validation and train. Something ...
hh32's user avatar
  • 2,732
81 votes

Train/Test/Validation Set Splitting in Sklearn

There is a great answer to this question over on SO that uses numpy and pandas. The command (see the answer for the discussion): ...
0_0's user avatar
  • 955
60 votes
Accepted

What is the difference between bootstrapping and cross-validation?

Both cross validation and bootstrapping are resampling methods. bootstrap resamples with replacement (and usually produces new "surrogate" data sets with the same number of cases as the original ...
cbeleites unhappy with SX's user avatar
59 votes
Accepted

Merging multiple data frames row-wise in PySpark

Stolen from: https://stackoverflow.com/questions/33743978/spark-union-of-multiple-rdds Outside of chaining unions this is the only way to do it for DataFrames. ...
Jan van der Vegt's user avatar
47 votes
Accepted

Why use both validation set and test set?

Let's assume that you are training a model whose performance depends on a set of hyperparameters. In the case of a neural network, these parameters may be for instance the learning rate or the number ...
Pablo Suau's user avatar
  • 1,787
45 votes
Accepted

How to use the output of GridSearch?

Decided to go away and find the answers that would satisfy my question, and write them up here for anyone else wondering. The .best_estimator_ attribute is an instance of the specified model type, ...
Dan Carter's user avatar
  • 1,732
42 votes

Train/Test/Validation Set Splitting in Sklearn

Adding to @hh32's answer, while respecting any predefined proportions such as (75, 15, 10): ...
Andrei Florea's user avatar
38 votes
Accepted

How does the validation_split parameter of Keras' fit function work?

You actually would not want to resample your validation set after each epoch. If you did this your model would be trained on every single sample in your dataset and thus this will cause overfitting. ...
JahKnows's user avatar
  • 8,876
27 votes
Accepted

Cross validation Vs. Train Validate Test

If k-fold cross-validation is used to optimize the model parameters, the training set is split into k parts. Training happens k times, each time leaving out a different part of the training set. ...
Louic's user avatar
  • 512
25 votes
Accepted

How to calculate the fold number (k-fold) in cross validation?

The number of folds is usually determined by the number of instances contained in your dataset. For example, if you have 10 instances in your data, 10-fold cross-validation wouldn't make sense. $k$-...
JahKnows's user avatar
  • 8,876
19 votes
Accepted

Cross Validation in Keras

From the Keras documentation, you can load the data into Train and Test sets like this: (X_train, y_train), (X_test, y_test) = mnist.load_data() As for cross ...
Christian Safka's user avatar
18 votes

Why use both validation set and test set?

The test set and cross validation set have different purposes. If you drop either one, you lose its benefits: The cross validation set is used to help detect over-fitting and to assist in hyper-...
Neil Slater's user avatar
  • 28.9k
17 votes

Merging multiple data frames row-wise in PySpark

Sometime, when the dataframes to combine do not have the same order of columns, it is better to df2.select(df1.columns) in order to ensure both df have the same ...
Wong Tat Yau's user avatar
16 votes
Accepted

How to choose a classifier after cross-validation?

You do cross-validation when you want to do any of these two things: Model Selection Error Estimation of a Model Model selection can come in different scenarios: Selecting one algorithm vs others ...
Javierfdr's user avatar
  • 1,490
14 votes
Accepted

Why you shouldn't upsample before cross validation

To see clearly why the procedure of upsampling before CV is mistaken and it leads to data leakage and other undesired consequences, it is useful to imagine first the simpler "baseline" case, ...
desertnaut's user avatar
  • 1,988
13 votes

How to calculate the fold number (k-fold) in cross validation?

Depends on how much CPU juice you are willing to afford for the same. Having a lower K means less variance and thus, more bias, while having a higher K means more variance and thus, and lower bias. ...
Dawny33's user avatar
  • 8,296
13 votes
Accepted

Cross validation for highly imbalanced data with undersampling

You should always do your evaluation of model performance on data that has not been over/undersampled. You can setup a pipeline with scikit-learn to perform your undersampling on the training set and ...
Wes's user avatar
  • 682
12 votes
Accepted

How to estimate GridSearchCV computing time?

You could fit your model/pipeline (with default parameters) to your data once and see how long it takes to train. Then you would multiply that by how many times you want to train the model through ...
Djib2011's user avatar
  • 7,978
12 votes
Accepted

What is GridSearchCV doing after it finishes evaluating the performance of parameter combinations that takes so long?

Yep I figured it out. The answer is that by default GridSearchCV's last act is to expose the API of the estimator object you passed so that you can directly call things like ...
Dan Scally's user avatar
  • 1,754
11 votes

Train/Test/Validation Set Splitting in Sklearn

You can use train_test_split twice. I think this is most straightforward. ...
David Jung's user avatar
11 votes
Accepted

Validation vs. test vs. training accuracy. Which one should I compare for claiming overfit?

Which two accuracies I compare to see if the model is overfitting or not? You should compare the training and test accuracies to identify over-fitting. A training accuracy that is subjectively far ...
Esmailian's user avatar
  • 9,312
11 votes

Why is the k-fold cross-validation needed?

Given the randomization, it is unlikely that there will be a dramatic change from one run into the next one in the loop of the cross-validation. This assumption is wrong, it's true only if the ...
Erwan's user avatar
  • 25.3k
10 votes
Accepted

Can overfitting occur even with validation loss still dropping?

I am not sure if the validation set is balanced or not. You have a severe data imbalance problem. If you sample equally and randomly from each class to train your network, and then a percentage of ...
Bashar Haddad's user avatar
10 votes

Why GridSearchCV returns nan?

By default, GridSearchCV provides a score of nan when fitting the model fails. You can change that behavior and raise an error ...
Ben Reiniger's user avatar
  • 11.8k
9 votes
Accepted

Nested cross-validation and selecting the best regression model - is this the right SKLearn process?

Yours is not an example of nested cross-validation. Nested cross-validation is useful to figure out whether, say, a random forest or a SVM is better suited for your problem. Nested CV only outputs a ...
Ricardo Cruz's user avatar
  • 3,420
9 votes
Accepted

Which is first ? Tuning the parameters or selecting the model

You can tune parameters only if you have already trained the model, otherwise there is nothing to tune. However, i've also read that model selection shoud be done before tuning the parameters. ...
Yaroslaw Homenko's user avatar
9 votes
Accepted

Using Cross Validation technique for a CNN model

Question 1: Why do most CNN models not apply the cross-validation technique? $k$-fold cross-validation is often used for simple models with few parameters, models with simple hyperparameters and ...
MachineLearner's user avatar
8 votes

How to choose a classifier after cross-validation?

No. You don't select any of the k classifiers built during k-fold cross-validation. First of all, the purpose of cross-validation is not to come up with a predictive model, but to evaluate how ...
tuomastik's user avatar
  • 1,183
8 votes
Accepted

How to implement Python's MLPClassifier with gridsearchCV?

A tuple of the form $(i_1, i_2, i_3, ... , i_n)$ gives you a network with $n$ hidden layers, where $i_k$ gives you the number of neurons in the $k$th hidden layer. If you want three hidden layers ...
oW_'s user avatar
  • 6,347
8 votes
Accepted

Stratify on regression

I have done this before and didn't find a default implementation - the StratifiedKFold and RepeatedStratifiedKFold are only documented to work with classes. The way I ended up doing it was not quite ...
n1k31t4's user avatar
  • 14.9k

Only top scored, non community-wiki answers of a minimum length are eligible