Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

Filter by
Sorted by
Tagged with
1
vote
0answers
10 views

How to apply oversampling when doing Leave-One-Group-Out cross validation?

I am working on an imbalanced data for classification and I tried to use SMOTE previously to oversampling the training data. However, this time I think I need to use a leave-on group out (LOGO) cross-...
0
votes
2answers
22 views

Is this the way to obtain the same individuals for x_test and y_test?

x_train, x_test = train_test_split(x, test_size = 0.3,random_state=250) y_train, y_test = train_test_split(y, test_size = 0.3,random_state=250) Is this the way to ...
0
votes
1answer
11 views

Training Deep learning and validation loss

I'm trying to replicate result of a paper. The paper is a U-net for De-noising of some images. So basically I have a simple U-net that I give noisy data as input and have denoised data as the wanted ...
0
votes
0answers
17 views

How to put KerasClassifier, Hyperopt and Sklearn cross-validation together

I am performing a hyperparameter tuning optimization (hyperopt) tasks with sklearn on a Keras models. I am trying to optimize KerasClassifiers using the Sklearn cross-validation, Some code follows: <...
0
votes
0answers
9 views

How to check performance of a model on a test set?

I have transformed my training set (predictor variables) using step_YeoJohnson for satisfying the assumptions of model. But now how do I run my model on test set which is not transformed and has ...
0
votes
1answer
17 views

Bias and variance in the model o in the predictions?

This topic confuses me. In the literature or articles, when talking about bias and variance in automatic learning, specifically in cross-validation, do they refer to the high bias (underfitting) and ...
2
votes
1answer
69 views

Training with a subset of data: relationship between subset size and training metric?

I have a not-quite linear regression problem which I am investigating. The data set is fairly large, with ~6000 samples and ~2100 features. By performing 5-fold cross-validation on different sized ...
1
vote
1answer
24 views

Determining threshold in an area with very few samples of positive label

I have a binary classification task where I want to either keep or discard samples. I have about a million samples, and about 1% should be kept. I want to discard as much as possible, but discarding ...
1
vote
0answers
27 views

Using pipelines with a cross validation of several models in scikit-learn

Is there a simple way to cross-validate several models using sklearn pipelines?
5
votes
1answer
50 views

Why could an overfitted CNN model have a higher validation accuracy?

I am currently training a CNN model by using cifar10 images (50000 for training, another 10000 for validation). I plot training loss, validation loss and accuracy against training iteration: I am ...
1
vote
1answer
32 views

High train and val results. Bad test and predict results

For my thesis project I've been trying to make a CNN for some challenging data. There's four classes with the following amount of images respectively [410, 410, 269, 206] = 1,295 total. Now I know ...
2
votes
3answers
192 views

In k-fold-cross-validation, why do we compute the mean of the metric of each fold

In k-fold-cross-validation, the "correct" scheme seem to compute the metric (say the accuracy) for each fold, and then return the mean as the final metric. Source : https://scikit-learn.org/stable/...
0
votes
1answer
26 views

Ideal score of a model on training and cross validation data

The question is little bit broad, but I could not find any concrete explanation anywhere, hence decided to ask the experts here. I have trained a classifier model for binary classification task. Now ...
1
vote
0answers
31 views

Is applying simultaneous K Fold Cross Validation and Drop out possible?

Well, it might seem ridiculous but I was just thinking whether it is possible to have these two methods simultaneously or not. I ran the code and faced an error, but in theory it doesn't seem ...
1
vote
2answers
40 views

Is it a good idea to tune the number of folds for cross validation when tuning hyperparameters of RF

I'm new to data science. I'm trying to get the best model for Random Forest. Unfortunately, I'm not sure if my idea can produce a good generalized model. 1) I have split data to TrainingSet (70%) and ...
2
votes
3answers
134 views

Cross validation Vs. Train Validate Test

I have a doubt regarding the cross validation approach and train-validation-test approach. I was told that I can split a dataset into 3 parts: Train: we train the model. Validation: we validate and ...
0
votes
1answer
24 views

K-Fold and Random Subsampling (RSS) Dataset generation?

Let say if I have a large dataset of 300k instances with 200 features, I want to reduce its size. Can I apply K-Fold technique to the 200 features then the trimmed dataset are applied with RSS to trim ...
0
votes
1answer
21 views

Applying Hold-out and CV technique

I have a methodology question: are hold-out and CV generalization-optimization techniques mutually exclusive? It gets really confusing to me at times, because in the most recent project I have been ...
0
votes
1answer
30 views

Am I using GridSearch correctly or do I need to use all data for cross validation?

I'm working with a dataset that has 400 observations, 34 features and quite a few outliers, some of them extreme. Given the nature of my data, these need to be in the model. I started by doing a 75-...
2
votes
1answer
40 views

Cross-validation for model comparison: use the same folds?

Let's say we have model M1 and model M2 that we want to compare. When we do 5-fold (say) cross validation, would the correct method to be to partition the data into F1, F2, F3, F4, and F5 and then run ...
2
votes
1answer
29 views

Help with understanding cross-validation

My understanding of cross-validation is that we divide our data set into parts 1-k, then use part 1 as a validation set and parts 2-k as a training set, then use part 2 as a validation set and the ...
3
votes
1answer
84 views

Model comparison with CV using standard error

Discovering the ML world with sklearn, I'm testing a large panel of models onto my dataset. This is for learning purpose but also for work so I want the final model ...
2
votes
1answer
142 views

Which model to chose based on learning curve

I trained my model using different regression techniques, and I'm not sure which model to choose based on the learning curve. 1) Should I choose Lasso, since train and CV converge at the end 2) ...
3
votes
1answer
75 views

Hyper parameters tuning XGBClassifier

I am working on a highly imbalanced dataset for a competition. The training data shape is : (166573, 14) ...
3
votes
1answer
71 views

Can we use k fold Cross Validation without any extra (excluded) Test Set?

I have seen this in two Papers: The authors use 10 fold cross validation, and then present the results from this validation or even odder the results from the best Fold as their modelling Result. ...
0
votes
0answers
34 views

Error while using lightGBM's cv() function for a regression problem

I am trying to use lightGBM's cv() function for tuning my model for a regression problem. My main model is ...
3
votes
1answer
269 views

XGBRegressor hyperparameter optimization using xgb cv function

I am trying to optimize hyper parameters of XGBRegressor using xgb's cv function and bayesian optimization (using hyperopt package). Here is the piece of code I am using for the cv part. ...
0
votes
1answer
38 views

Is the role of the validation set in a deep learning network is only for Early Stopping?

In the "deep learning crash course" given by Leo Isikdogan in lecture 4 https://www.youtube.com/watch?v=ms-Ooh9mjiE&list=PLWKotBjTDoLj3rXBL-nEIPRN9V3a9Cx07&index=4 Overfitting, Underfitting, ...
0
votes
0answers
16 views

Getting parameters of the best model with crossvalidation in with SparkMLLib

I am having trouble accessing the parameters of estimators of model in SparkMLlib. More precisely my problem is: I have a logistic regression model for which I want to find the best regularization ...
2
votes
0answers
39 views

Is AUC a good metric for evaluating the performance of a multi-class classification?

Considering the definition of AUC (Area Under Curve), is that a reliable performance metric for a multi-class (30-40 classes) classification problem?
3
votes
0answers
105 views

Why does CV yield lower score?

My training accuracy was better than my test accuracy, hence I thought my model was over-fitted and tried Cross-validation. The model further degraded. Is that my input data need to be sanitised ...
1
vote
0answers
60 views
1
vote
1answer
96 views

How to determine number of leaves in decision tree analysis?

Would be grateful if some expert on the forum can help me understand how to decide optimum number of leaves in a decision tree analysis. I am using SAS and if I supply leaves=6 in my model then miss-...
1
vote
0answers
97 views

Plotting ROC curve in cross validation using Matlab perfcurve

I have the following code for a binary classifying using SVM, and 10 cross-validation, ...
7
votes
2answers
805 views

Validation vs. test vs. training accuracy. Which one should I compare for claiming overfit?

I have read on the several answers here and on the Internet that cross-validation helps to indicate that if the model will generalize well or not and about overfitting. But I am confused that which ...
0
votes
1answer
19 views

Choose CNN architecture first, then optimize parameters - validation vs test performance to pick architecture?

I am doing a few experiments on medical data. I am about to transfer learn the pretrained networks for my problem. Firstly, I have to pick a network architecture. Secondly, I would like to optimize ...
0
votes
1answer
34 views

Evaluating the test set

Please find attached a part of the code which explains what I'm trying to do. Essentially I'm trying to predict the sales of supermarket stores. Im using RandomForestRegressor for this and have ...
0
votes
0answers
41 views

What to do when Kfold is not enough?

I have a dataset made of roughly 100 time-series and my final goal is to obtain a classification of each point (detection problem). To do so I have labels so I decided to use an XGB model to perform ...
-1
votes
1answer
510 views

k-fold cross validation in keras for regression using sklearn [closed]

I am using a wrapper to use sklearn k-fold cross-validation with keras for a regression problem with ANN. but the accuracies i get look very weird. It has worked fine for a classification problem. I ...
1
vote
1answer
15 views

Change rate of cross validation data, after training

Say we have N of labeled data, and we need to take some part for the cross validation (we will skip test part for this case). We ...
3
votes
1answer
127 views

Overfitting - how to detect it and reduce it?

I have a side project where I am doing credit scoring using R (sample size around 16k for train data and 4k for test data, and also another two 20k data batches for out-of-time validation) with ...
1
vote
1answer
54 views

80-20 better than full dataset for LightGBM

Recently I have been using LightGBM as regressor in order to predict, on a dataset of 20 thousand observations and 40 variables. I have two modes, 1) Production and 2) Testing. The first one just ...
0
votes
0answers
52 views

How can I recognise if I can improve a random forest model by adding features

I want to tune a random forest model with caret package. I'm tuning it with cross-validation to prevent overfitting and resulted cross-validation accuracy is very ...
0
votes
0answers
36 views

Cross-validation and out-of-bag bootstrap applications

I have a question regarding steps on which a specific resample method should be used in general. As far as I know: out-of-bag bootstrap is the resample method with replacement, which has lower ...
2
votes
1answer
73 views

Should I oversample my validation data to get better F1 score and PRC?

I am currently working with a dataset that is imbalanced, about 30k rows * 14 features (just for you know), and 99.5% of the data is labeled 0. Since the model is strongly imbalanced I decided to use ...
1
vote
0answers
18 views

Performance diagnostics in mxnet gluon (e.g. plotting training vs validation loss over time)?

Tensorflow has tensorboard, is there any recommended way to plot classification error/loss over time in mxnet?
2
votes
1answer
46 views

Is splitting the data set into train and validation applicable in unsupervised learning?

I am having a tough time implementing all the steps of setting up support vector machine (SVM) for unsupervised learning. My data set is labelled but for educational purposes I am learning ...
1
vote
1answer
205 views

Target encoding with cross validation

I am trying to understand this way of target (mean/impact/likelihood) encoding using (two-level) cross validation. It's taking mean value of y. But not plain mean, but in cross-validation within ...
3
votes
1answer
308 views

PCA, SMOTE and cross validation- how to combine them together?

I was reading a lot recently about PCA and cross validation and it seems that the majority call it malpractice to do PCA before cross validation. I would also like to perform SMOTE, but there is a ...