173

You could just use sklearn.model_selection.train_test_split twice. First to split to train, test and then split train again into validation and train. Something like this: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1) X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.25, ...


65

There is a great answer to this question over on SO that uses numpy and pandas. The command (see the answer for the discussion): train, validate, test = np.split(df.sample(frac=1), [int(.6*len(df)), int(.8*len(df))]) produces a 60%, 20%, 20% split for training, validation and test sets.


53

Stolen from: https://stackoverflow.com/questions/33743978/spark-union-of-multiple-rdds Outside of chaining unions this is the only way to do it for DataFrames. from functools import reduce # For Python 3.x from pyspark.sql import DataFrame def unionAll(*dfs): return reduce(DataFrame.unionAll, dfs) unionAll(td2, td3, td4, td5, td6, td7, td8, td9, ...


42

Both cross validation and bootstrapping are resampling methods. bootstrap resamples with replacement (and usually produces new "surrogate" data sets with the same number of cases as the original data set). Due to the drawing with replacement, a bootstrapped data set may contain multiple instances of the same original cases, and may completely omit other ...


40

Decided to go away and find the answers that would satisfy my question, and write them up here for anyone else wondering. The .best_estimator_ attribute is an instance of the specified model type, which has the 'best' combination of given parameters from the param_grid. Whether or not this instance is useful depends on whether the refit parameter is set to ...


38

Let's assume that you are training a model whose performance depends on a set of hyperparameters. In the case of a neural network, these parameters may be for instance the learning rate or the number of training iterations. Given a choice of hyperparameter values, you use the training set to train the model. But, how do you set the values for the ...


28

You actually would not want to resample your validation set after each epoch. If you did this your model would be trained on every single sample in your dataset and thus this will cause overfitting. You want to always split your data before the training process and then the algorithm should only be trained using the subset of the data for training. The ...


21

The number of folds is usually determined by the number of instances contained in your dataset. For example, if you have 10 instances in your data, 10-fold cross-validation wouldn't make sense. $k$-fold cross validation is used for two main purposes, to tune hyper parameters and to better evaluate the performance of a model. In both of these cases selecting ...


18

From the Keras documentation, you can load the data into Train and Test sets like this: (X_train, y_train), (X_test, y_test) = mnist.load_data() As for cross validation, you could follow this example from here. from sklearn.model_selection import StratifiedKFold def load_data(): # load your data using this function def create model(): # create ...


18

Adding to @hh32's answer, while respecting any predefined proportions such as (75, 15, 10): train_ratio = 0.75 validation_ratio = 0.15 test_ratio = 0.10 # train is now 75% of the entire data set # the _junk suffix means that we drop that variable completely x_train, x_test, y_train, y_test = train_test_split(dataX, dataY, test_size=1 - train_ratio) # test ...


16

Yes, you are correct that the dominant difference between the area under the curve of a receiver operator characteristic curve (ROC-AUC) and the area under the curve of a Precision-Recall curve (PR-AUC) lies in its tractability for unbalanced classes. They are very similar and have been shown to contain essentially the same information, however PR curves ...


16

The test set and cross validation set have different purposes. If you drop either one, you lose its benefits: The cross validation set is used to help detect over-fitting and to assist in hyper-parameter search. The test set is used to measure the performance of the model. You cannot use the cross validation set to measure performance of your model ...


15

Sometime, when the dataframes to combine do not have the same order of columns, it is better to df2.select(df1.columns) in order to ensure both df have the same column order before the union. import functools def unionAll(dfs): return functools.reduce(lambda df1,df2: df1.union(df2.select(df1.columns)), dfs) Example: df1 = spark.createDataFrame([[1,1],...


15

You do cross-validation when you want to do any of these two things: Model Selection Error Estimation of a Model Model selection can come in different scenarios: Selecting one algorithm vs others for a particular problem/dataset Selecting hyper-parameters of a particular algorithm for a particular problem/dataset (please notice that if you are both ...


15

If k-fold cross-validation is used to optimize the model parameters, the training set is split into k parts. Training happens k times, each time leaving out a different part of the training set. Typically, the error of these k-models is averaged. This is done for each of the model parameters to be tested, and the model with the lowest error is chosen. The ...


13

You should always do your evaluation of model performance on data that has not been over/undersampled. You can setup a pipeline with scikit-learn to perform your undersampling on the training set and then evaluate on the non-undersampled fold of data for each iteration as you described.


10

I am not sure if the validation set is balanced or not. You have a severe data imbalance problem. If you sample equally and randomly from each class to train your network, and then a percentage of what you sampled is used to validate your network , this means that you train and validate using balanced data set. In the testing you used imbalanced database. ...


10

Depends on how much CPU juice you are willing to afford for the same. Having a lower K means less variance and thus, more bias, while having a higher K means more variance and thus, and lower bias. Also, one should keep in mind the computational costs for the different values. High K means more folds, thus higher computational time and vice versa. So, one ...


10

You could fit your model/pipeline (with default parameters) to your data once and see how long it takes to train. Then you would multiply that by how many times you want to train the model through grid search. E.g. suppose you want to use a grid search to select the hyperparameters a, b and c of your pipeline. params = {'a': [1, 2, 3, 4, 5], 'b': [...


10

Yep I figured it out. The answer is that by default GridSearchCV's last act is to expose the API of the estimator object you passed so that you can directly call things like .predict() or .score() on the GridSearchCV object itself. It does this by retraining the estimator against the best parameters it found during cross validation. If you want to skip this ...


10

To see clearly why the procedure of upsampling before CV is mistaken and it leads to data leakage and other undesired consequences, it is useful to imagine first the simpler "baseline" case, where we simply upsample (i.e. create duplicate samples) without SMOTE. The first reason why such a procedure is invalid is that, this way, some of the ...


9

You can use train_test_split twice. I think this is most straightforward. X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=1) X_train, X_val, y_train, y_val = train_test_split( X_train, y_train, test_size=0.25, random_state=1) In this way, train, val, test set will be 60%, 20%, 20% of the dataset respectively.


9

Which two accuracies I compare to see if the model is overfitting or not? You should compare the training and test accuracies to identify over-fitting. A training accuracy that is subjectively far higher than test accuracy indicates over-fitting. Here, "accuracy" is used in a broad sense, it can be replaced with F1, AUC, error (increase becomes decrease, ...


8

If you have an adequate number of samples and want to use all the data, then k-fold cross-validation is the way to go. Having ~1,500 seems like a lot but whether it is adequate for k-fold cross-validation also depends on the dimensionality of the data (number of attributes and number of attribute values). For example, if each observation has 100 attributes, ...


8

By default random forest picks up 2/3rd data for training and rest for testing for regression and almost 70% data for training and rest for testing during classification.By principle since it randomizes the variable selection during each tree split it's not prone to overfit unlike other models.However if you want to use CV using nfolds in sklearn you can ...


8

Yours is not an example of nested cross-validation. Nested cross-validation is useful to figure out whether, say, a random forest or a SVM is better suited for your problem. Nested CV only outputs a score, it does not output a model like in your code. This would be an example of nested cross validation: from sklearn.datasets import load_boston from ...


8

No. You don't select any of the k classifiers built during k-fold cross-validation. First of all, the purpose of cross-validation is not to come up with a predictive model, but to evaluate how accurately a predictive model will perform in practice. Second of all, for the sake of argument, let's say you were to use k-fold cross-validation with k=10 to find ...


8

You can tune parameters only if you have already trained the model, otherwise there is nothing to tune. However, i've also read that model selection shoud be done before tuning the parameters. Before tuning you need to do some kind of pre-processing before tuning the parameters. Usually your pipeline will consist of: Get Data and Clean It. Do some EDA ( ...


8

Question 1: Why do most CNN models not apply the cross-validation technique? $k$-fold cross-validation is often used for simple models with few parameters, models with simple hyperparameters and additionally the models are easy to optimize. Typical examples are linear regression, logistic regression, small neural networks and support vector machines. For a ...


7

No, he actually says the opposite: One final note: I should say that in the machine learning as of this practice today, there are many people that will do that early thing that I talked about, and said that, you know...​ Then he says (the "early thing" he talked about): selecting your model as a test set and then using the same test set to report the ...


Only top voted, non community-wiki answers of a minimum length are eligible