Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

Filter by
Sorted by
Tagged with
158
votes
14answers
267k views

Train/Test/Validation Set Splitting in Sklearn

How could I randomly split a data matrix and the corresponding label vector into a X_train, X_test, ...
40
votes
3answers
42k views

What is the difference between bootstrapping and cross-validation?

I used to apply K-fold cross-validation for robust evaluation of my machine learning models. But I'm aware of the existence of the bootstrapping method for this purpose as well. However, I cannot see ...
36
votes
2answers
84k views

How does the validation_split parameter of Keras' fit function work?

Validation-split in Keras Sequential model fit function is documented as following on https://keras.io/models/sequential/ : validation_split: Float between 0 and 1. Fraction of the training data ...
32
votes
2answers
30k views

Why use both validation set and test set?

Consider a neural network: For a given set of data, we divide it into training, validation and test set. Suppose we do it in the classic 60:20:20 ratio, then we prevent overfitting by validating the ...
31
votes
6answers
152k views

Merging multiple data frames row-wise in PySpark

I have 10 data frames pyspark.sql.dataframe.DataFrame, obtained from randomSplit as ...
31
votes
2answers
71k views

How to use the output of GridSearch?

I'm currently working with Python and Scikit learn for classification purposes, and doing some reading around GridSearch I thought this was a great way for optimising my estimator parameters to get ...
29
votes
3answers
51k views

Does modeling with Random Forests require cross-validation?

As far as I've seen, opinions tend to differ about this. Best practice would certainly dictate using cross-validation (especially if comparing RFs with other algorithms on the same dataset). On the ...
26
votes
2answers
39k views

How to calculate the fold number (k-fold) in cross validation?

I am confused about how I choose the number of folds (in k-fold CV) when I apply cross validation to check the model. Is it dependent on data size or other parameters?
15
votes
2answers
11k views

Can overfitting occur even with validation loss still dropping?

I have a convolutional + LSTM model in Keras, similar to this (ref 1), that I am using for a Kaggle contest. Architecture is shown below. I have trained it on my labeled set of 11000 samples (two ...
14
votes
3answers
14k views

Cross validation Vs. Train Validate Test

I have a doubt regarding the cross validation approach and train-validation-test approach. I was told that I can split a dataset into 3 parts: Train: we train the model. Validation: we validate and ...
13
votes
3answers
13k views

How to choose a classifier after cross-validation?

When we do k-fold cross validation, should we just use the classifier that has the highest test accuracy? What is generally the best approach in getting a classifier from cross validation?
12
votes
2answers
10k views

Cross-validation: K-fold vs Repeated random sub-sampling

I wonder which type of model cross-validation to choose for classification problem: K-fold or random sub-sampling (bootstrap sampling)? My best guess is to use 2/3 of the data set (which is ~1000 ...
12
votes
2answers
11k views

K-fold cross validation when using fit_generator and flow_from_directory() in Keras

I am using flow_from_directory() and fit_generator in my deep learning model, and I want to use cross validation method to train ...
11
votes
2answers
14k views

Validation vs. test vs. training accuracy. Which one should I compare for claiming overfit?

I have read on the several answers here and on the Internet that cross-validation helps to indicate that if the model will generalize well or not and about overfitting. But I am confused that which ...
11
votes
4answers
9k views

Overfitting/Underfitting with Data set size

In the below graph, x-axis => Data set Size y-axis => Cross validation Score Red line is for Training Data Green line is for Testing Data In a tutorial that I'm referring to, the author says ...
11
votes
2answers
3k views

What is the proper way to use early stopping with cross-validation?

I am not sure what is the proper way to use early stopping with cross-validation for a gradient boosting algorithm. For a simple train/valid split, we can use the valid dataset as the evaluation ...
10
votes
1answer
5k views

Cross validation for highly imbalanced data with undersampling

In my problem, I am dealing with a highly imbalanced data set, say for every positive class there are 10000 negative one. A normal starting method to train a model is to undersample the data. In this ...
10
votes
4answers
2k views

Will cross validation performance be an accurate indication for predicting the true performance on an independent data set?

I feel that this question is related to the theory behind cross-validation. I present my empirical finding here and wrote a question related to the theory of cross-validation at there. I have two ...
9
votes
2answers
42k views

Cross Validation in Keras

Suppose I would like to train and test the MNIST dataset in Keras. The required data can be loaded as follows: ...
9
votes
3answers
10k views

Nested cross-validation and selecting the best regression model - is this the right SKLearn process?

If I understand correctly, nested-CV can help me evaluate what model and hyperparameter tuning process is best. The inner loop (GridSearchCV) finds the best ...
9
votes
2answers
14k views

Using Cross Validation technique for a CNN model

I am working on a CNN model. As always, I used batches with epochs to train my model. When it completed training and validation, finally I used a test set to measure the model performance and generate ...
9
votes
4answers
3k views

Which is first ? Tuning the parameters or selecting the model

I've been reading about how we split our data into 3 parts; generally, we use the validation set to help us tune the parameters and the test set to have an unbiased estimate on how well does our model ...
9
votes
1answer
2k views

Stratify on regression

I have worked in classification problems, and stratified cross-validation is one of the most useful and simple techniques I've found. In that case, what it means is to build a training and validation ...
9
votes
1answer
4k views

How to approach the numer.ai competition with anonymous scaled numerical predictors?

Numer.ai has been around for a while now and there seem to be only few posts or other discussions about it on the web. The system has changed from time to time and the set-up today is the following: ...
8
votes
5answers
2k views

Time-series grouped cross-validation

I have data with the following structure: ...
8
votes
2answers
19k views

How to estimate GridSearchCV computing time?

If I know the time of a given validation with set values, can I estimate the time GridSearchCV will take for n values I want to cross-validate?
8
votes
6answers
2k views

Which cross-validation type best suits to binary classification problem

Data set looks like: 25000 observations up to 15 predictors of different types: numeric, multi-class categorical, binary target variable is binary Which cross validation method is typical for this ...
8
votes
3answers
8k views

What needs to be done to make n_jobs work properly on sklearn? in particular on ElasticNetCV?

The constructor of sklearn.linear_model.ElasticNetCV takesn_jobs as an argument. Quoting the documentation here n_jobs: int, ...
8
votes
1answer
8k views

How fbprophet cross validation works

I am facing some issues to understand how cross_validation function works in fbprophet packages. I have a time series of 68 days (only business days) grouped by 15min and a certain metric : 00:00 ...
8
votes
2answers
680 views

Machine Learning models in production environment

Lets say a Model was trained on date $dt1$ using the available labeled data, split into training and test i.e $train_{dt1}$, $test_{dt1}$. This model is then deployed in production and makes ...
8
votes
1answer
3k views

Why k-fold cross validation (CV) overfits? Or why discrepancy occurs between CV and test set?

Recently, I was working on a project and found my cross-validation error rate very low, but the testing set error rate very high. This might indicate that my model is overfitting. Why does my cross-...
7
votes
1answer
8k views

When do I have to use aucPR instead of auROC? (and vice versa)

I'm wondering if sometimes, to validate a model, it's not better to use aucPR instead of aucROC? Do these cases only depend on the "domain & business understanding" ? Especially, I'm thinking ...
7
votes
1answer
1k views

Why you shouldn't upsample before cross validation

I have an imbalanced dataset and I am trying different methods to address the data imbalance. I found this article that explains the correct way to cross-validate when oversampling data using SMOTE ...
7
votes
1answer
3k views

What is GridSearchCV doing after it finishes evaluating the performance of parameter combinations that takes so long?

I'm running GridSearchCV to tune some parameters. For example: ...
7
votes
2answers
5k views

Is there a way of performing stratified cross validation using xgboost module in python?

I am training and predicting on the same data-set, but I want to perform 10-fold cross-validation and predict on the left out fold and thus predict on the whole data set. How can I do this? The ...
7
votes
1answer
102 views

Why use mean revenue in a split test?

I asked a data science question regarding how to decide on the best variation of a split test on the Statistics section of StackExchange. I hope I will have better luck here. The question is basically,...
7
votes
4answers
5k views

Log loss vs accuracy for deciding between different learning rates?

While model tuning using cross validation and grid search I was plotting the graph of different learning rate against log loss and accuracy separately. Log loss When I used log loss as score in ...
6
votes
3answers
3k views

Linear kernel in SVM performing much worse than RBF or Poly

When trying to train a SVM on some Kaggle data, I have encountered a situation where the linear kernel fails to give any results. This doesn't make sense to me because the RBF kernel works just fine, ...
6
votes
2answers
199 views

ground truth fit is worse than cross validated fit on noisy data?

I am having these weird results when playing around with cross-validation that I would greatly appreciate having any comments. Briefly, I have a lower mean squared error (MSE) when doing regression (...
5
votes
3answers
2k views

ICD-10 codes in Machine Learning

Can anyone provide specific techniques with using ICD-10 codes in Machine Learning? I have usually used a simply approach of creating multiple binary column representing ICD-10 codes… which can get ...
5
votes
2answers
276 views

Why is 10 considered the default value for k-fold cross-validation?

I understand very well what k-fold cross-validation is. In my studies, and at work, I've always heard something along the lines of: We most often use k=10 ...
5
votes
2answers
28k views

How to implement Python's MLPClassifier with gridsearchCV?

I am trying to implement Python's MLPClassifier with 10 fold cross-validation using gridsearchCV function. Here is a chunk of my code: ...
5
votes
2answers
6k views

Cross validation when training neural network?

The standard setup when training a neural network seems to be to split the data into train and test sets, and keep running until the scores stop improving on the test set. Now, the problem: there is ...
5
votes
1answer
2k views

On coursera what exactly does Andrew Ng say in videos Lectures 60 & 61 of machine learning?

Model Selection and Train/Validation/Test Sets - Stanford University | Coursera: At 10:59~11:10 One final note: I should say that in the machine learning as of this practice today, there aren't ...
5
votes
3answers
3k views

Should you use random state or random seed in machine learning models?

I'm starting to study machine learning. All the examples I saw, the person that created the ML model used a random state or a random seed to stop the randomness of the process. But, in real life, when ...
5
votes
2answers
16k views

Using K-fold cross-validation in Keras on the data of my model

I would like to use K-fold cross-validation on my data of my model. My codes in Keras is : ...
5
votes
1answer
2k views

Can we use k fold Cross Validation without any extra (excluded) Test Set?

I have seen this in two Papers: The authors use 10 fold cross validation, and then present the results from this validation or even odder the results from the best Fold as their modelling Result. ...
5
votes
2answers
3k views

Cross Validation how to determine when to Early Stop?

When using "K-Fold Cross Validtion" for Neural Net, do we: Pick and save initial weights of the network randomly (let's call it $W_0$) Split data into $N$ equal chunks Train model on $N-1$ chunks, ...
5
votes
2answers
8k views

Overfitting XGBoost

I try to classify data from a dataset of 315 lines and 17 (real data) features (315x17). The target value is either "good" or "bad" (binary classification). I used XGBoost to classify these data, ...
5
votes
1answer
5k views

Which is better: Out of Bag (OOB) or Cross-Validation (CV) error estimates?

I have seen other posts in this forum but didn't find any convincing answer. Random Forest has an another way of tuning hyperparameter via OOB by design. OOB and CV are not the same as OOB error is ...

1
2 3 4 5
12