Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

Filter by
Sorted by
Tagged with
1
vote
1answer
29 views

Hyperparameter tuning and cross validation

I have some confusion about proper usage of cross-validation to tune hyperparameters and evaluate estimator performance and generalizeability. As I understand it, this would be the process you ...
0
votes
0answers
18 views

Value Error: MSLE & CrossVal

I'm trying to run cross validation with mean squared log error with sklearn and getting the following error message: ...
0
votes
1answer
20 views

clarification on train, test and val and how to use/implement it

So far I think I understood the differences between the training, test and validation set. Basically it is like in this image: Training set: The data where the model is trained on Validation set: ...
0
votes
1answer
16 views

Drawing validation set from test set

I am building a 3 neural network models on dataset that is already separated to train and test sets. From my analysis, I found that this dataset has values on test set which don't exist in the train ...
1
vote
1answer
20 views

K-fold-cross-validation if training dataset is much smaller than test dataset?

I'm a beginner in machine learning and I have a special case in which I have only a small training dataset of about 500 images and a test dataset of 10,000 images. Does it still make sense to do a 10-...
1
vote
1answer
12 views

How to construct validation set for time series for NN?

I would like to train my model with a validation set. As the data is a time series I have to use timeseriessplit: ...
2
votes
2answers
216 views

Advice and Ideas appreciated - Machine Learning one man project

I have a project where I am supposed to start from scratch and learn how machine Learning works. So far everything is working out better than expected but I feel as I am offered to many ways to choose ...
3
votes
2answers
113 views

XGBoost validation for number of trees

I have a simple Question: I am using XGBoost to classify some data: 1.) With 100 estimators I have following scores(roc_score): train_data : 98.5 validation_data : 97.2 2.) With 500 ...
0
votes
0answers
6 views

Recursive feature elimination on train data or complete dataset and dummy encoding

I am using RFE with logistic regression. I will also be doing cross validation with RFE (RFECV in sklearn) to get the optimum number of features. I am not sure whether to use RFECV on just train ...
0
votes
1answer
37 views

Cross-validation average score

I am using Repeated K-folds (RepeatedKFold(n_splits=10, n_repeats=10, random_state=999) from sklearn) to provide reliable scores for a linear regression on my ...
0
votes
0answers
7 views

Cross_validation is decreasing accuracy?

I have certain dataset to train a model. The dataset is not very small in size. First, I split the dataset into training and validation data using traintestsplit (80-20), train the model on training ...
1
vote
1answer
16 views

Cross validation Vs. Train Validate Test cont'd

Cross validation Vs. Train Validate Test I have one further question relating to @louic answer in the post above: "Training happens k times, each time leaving out a different part of the training ...
2
votes
3answers
53 views

Publish without validation score?

My mentor wants me to write and submit an academic paper reporting a predictive model, but without any validation score. Everything I have read in textbooks or the Internet says that this is wrong, ...
1
vote
1answer
27 views

Cross validation test and train errors

I came across this sort of flowchart: Below the flowchart, this is what appears: “Given a training set, cross-validation error is computed for each configuration of tuning parameters (λ,d). The ...
1
vote
0answers
37 views

Data leak when training on future data and testing on present [closed]

Given a time series dataset. Using simple train_test_split, then reversing the train to test and test to train i.e. using the future data to train and present data to test, where does one induce leak ...
1
vote
1answer
21 views

Validity of cross-validation for model performance estimation

When applying cross-validation for estimating the performance of a predictive model, the reported performance is usually the average performance over all the validation folds. As during this procedure,...
1
vote
1answer
30 views

What to do about non-responses to demographic survey questions?

Can anyone point me to some think pieces on what to do with census type data where at least some of the people surveyed do not self-identify a race/ethnicity, gender, or other demographic data? In ...
0
votes
1answer
16 views

Cross validation while preserving a column (not the target ) distribution

So i'm doing cross validation and then i'm predicting using all the data on a test set ( a hold-out set ). My hold-out set has the same ratio on a column than the train ( seems thats how the test set ...
0
votes
1answer
13 views

Interpreting a curve val_loss and loss in keras after training a model

I am having trouble understanding the curve val_loss and loss in keras after training my model. Can anyone help me understand ...
4
votes
1answer
58 views

sklearn's cross_validate does not work with catboost

I would like to use cross validation with catboost. Since I do not just want to use catboost but also sampling I am using a ...
1
vote
1answer
47 views

Oversampling only balances the training set, what about the testing set?

In a case of imbalanced data classification, I know that we only oversample the training set (to prevent data leakage from training to testing subsets), but what if there are no positive data points ...
2
votes
3answers
60 views

Should I perform cross validation only on the training set?

I am working with a dataset that I downloaded from Kaggle. The data set is already divided into two CSVs for Train and Test. I built a model using the training set because I imported the train CSV ...
2
votes
1answer
32 views

Significant drop from validation accuracy to test accuracy

I am more familiar with classification tasks, though I have been working on a regression problem. I was given a large training dataset (>70k samples) and an independently collected test set (~2k). I ...
0
votes
2answers
41 views

Does high accuracy metrics with small (but equally sampled) dataset means a good model?

I have been training my CNN with 200 images per class for a classification problem. There problem is a binary classification one. And with the amount of test data ( 25 per class) I am getting good ...
4
votes
1answer
37 views

help understanding nested cross validation

From what I read online, nested CV works as follows: I divide my whole data in k folds, where k-1 folds are the train set and one fold is the test set. There is the inner CV loop, where we may ...
1
vote
1answer
38 views

choosing classifiers

For what I read the 5x2cv t test is "a procedure for comparing the performance of two models (classifiers or regressors) that was proposed by Dietterich to address shortcomings in other methods such ...
0
votes
1answer
45 views

sckit-learn Cross validation and model retrain

I want to train a model and also perform cross validation in scikit-learn, If i want to access the model (For instance to see the parameter's selected and weights or to predict) i will need to fit it ...
3
votes
1answer
30 views

CrossValidation using glmnet and very high values of Lambda?

I am trying to run crossvalidation (folds=10) using glmnet library on my dataset. My outcome of interest is BMI and predictors include a set of clinical variables. My final goal is to use elastic-net ...
1
vote
1answer
30 views

Using of 100s of Binary features in regression model

I have 100s of columns with binary values [0, 1] plus some extra columns without binary values. I am trying to do regression model but the model performance is very low. For non-binary features, I ...
4
votes
1answer
143 views

Interpretability of RMSE and R squared scores on cross validation

I'm working on a regression problem with 30k rows in my dataset, decided to use XGBoost mainly to avoid processing data for a quick primitive model. And i noticed upon doing cross-validation that ...
2
votes
1answer
62 views

How to deploy ML models in production?

Have been learning Machine Learning concepts and doing hands-on past few months. Now got a bit different doubt than implementation of code. This is regarding measures to be considered while ...
1
vote
0answers
15 views

cross validation in time series

I am aware there are time series cross validation methods however I don't think there is one yet which solves my task. I want to know if my idea seems reasonable or if there are better solutions out ...
0
votes
1answer
13 views

Is it okay to use training data for validifying the trained model?

Currently, I have trained my model through 5-fold cross validation with very small amount of the sample (n=240). I used whole data set to train and got quite low performance in terms of accuracy, ...
2
votes
2answers
42 views

Best practice with cross validation

I have done a 10 fold Cross Validation on my data and have selected the best model from the results. With cross validation, I will have 10 models trained from different folds of the data. For the ...
0
votes
1answer
20 views

Splitting large multi class dataset using leave one out scheme into train and test

I am doing some supervised learning using neural networks, and i have a Targets array containing 1906 samples, which contain 664 unique values. min. count of each unique value==2, by design. Is there ...
1
vote
0answers
38 views

How to apply oversampling when doing Leave-One-Group-Out cross validation?

I am working on an imbalanced data for classification and I tried to use SMOTE previously to oversampling the training data. However, this time I think I need to use a leave-on group out (LOGO) cross-...
1
vote
2answers
29 views

Is this the way to obtain the same individuals for x_test and y_test?

x_train, x_test = train_test_split(x, test_size = 0.3,random_state=250) y_train, y_test = train_test_split(y, test_size = 0.3,random_state=250) Is this the way to ...
0
votes
1answer
13 views

Training Deep learning and validation loss

I'm trying to replicate result of a paper. The paper is a U-net for De-noising of some images. So basically I have a simple U-net that I give noisy data as input and have denoised data as the wanted ...
1
vote
0answers
38 views

How to put KerasClassifier, Hyperopt and Sklearn cross-validation together

I am performing a hyperparameter tuning optimization (hyperopt) tasks with sklearn on a Keras models. I am trying to optimize KerasClassifiers using the Sklearn cross-validation, Some code follows: <...
0
votes
0answers
10 views

How to check performance of a model on a test set?

I have transformed my training set (predictor variables) using step_YeoJohnson for satisfying the assumptions of model. But now how do I run my model on test set which is not transformed and has ...
0
votes
1answer
28 views

Bias and variance in the model o in the predictions?

This topic confuses me. In the literature or articles, when talking about bias and variance in automatic learning, specifically in cross-validation, do they refer to the high bias (underfitting) and ...
2
votes
1answer
75 views

Training with a subset of data: relationship between subset size and training metric?

I have a not-quite linear regression problem which I am investigating. The data set is fairly large, with ~6000 samples and ~2100 features. By performing 5-fold cross-validation on different sized ...
1
vote
1answer
27 views

Determining threshold in an area with very few samples of positive label

I have a binary classification task where I want to either keep or discard samples. I have about a million samples, and about 1% should be kept. I want to discard as much as possible, but discarding ...
1
vote
0answers
30 views

Using pipelines with a cross validation of several models in scikit-learn

Is there a simple way to cross-validate several models using sklearn pipelines?
5
votes
1answer
78 views

Why could an overfitted CNN model have a higher validation accuracy?

I am currently training a CNN model by using cifar10 images (50000 for training, another 10000 for validation). I plot training loss, validation loss and accuracy against training iteration: I am ...
1
vote
1answer
62 views

High train and val results. Bad test and predict results

For my thesis project I've been trying to make a CNN for some challenging data. There's four classes with the following amount of images respectively [410, 410, 269, 206] = 1,295 total. Now I know ...
2
votes
3answers
218 views

In k-fold-cross-validation, why do we compute the mean of the metric of each fold

In k-fold-cross-validation, the "correct" scheme seem to compute the metric (say the accuracy) for each fold, and then return the mean as the final metric. Source : https://scikit-learn.org/stable/...
0
votes
1answer
27 views

Ideal score of a model on training and cross validation data

The question is little bit broad, but I could not find any concrete explanation anywhere, hence decided to ask the experts here. I have trained a classifier model for binary classification task. Now ...
1
vote
0answers
33 views

Is applying simultaneous K Fold Cross Validation and Drop out possible?

Well, it might seem ridiculous but I was just thinking whether it is possible to have these two methods simultaneously or not. I ran the code and faced an error, but in theory it doesn't seem ...