Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

156 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
0answers
123 views

Repeated k-fold Cross Validation for time series data?

I have a relative small sample size (330 with 45 features) + it's time series data. I want to train my LightGBM regression model for best generalized RMSE score and want to use repeated CV. I use ...
4
votes
1answer
478 views

Cross-validation for model comparison: use the same folds?

Let's say we have model M1 and model M2 that we want to compare. When we do 5-fold (say) cross validation, would the correct method to be to partition the data into F1, F2, F3, F4, and F5 and then run ...
4
votes
1answer
239 views

Why does CV yield lower score?

My training accuracy was better than my test accuracy, hence I thought my model was over-fitted and tried Cross-validation. The model further degraded. Is that my input data need to be sanitised ...
4
votes
1answer
84 views

EM-ELM Cross validation

I know that cross validation is used to find the best hyperparameters that minimize the average error. For example, the number of neurons that minimize the average error of cross-validation is ...
3
votes
1answer
80 views

What parameters to use when normalising training, validation, and testing data?

I know a similar post was made here, but I wanted to ask some follow up questions. I am conducting a cross-validation search to find values of a set of hyper-parameters and need to normalise the data. ...
3
votes
0answers
76 views

SuperLearner Cross validation with iid time series

I created a number of ML models in R and I aim at combining them to form an ensemble. I learned about SuperLearner library which cross validates many models and returns the weight to each model in ...
3
votes
0answers
409 views

How to apply oversampling when doing Leave-One-Group-Out cross validation?

I am working on an imbalanced data for classification and I tried to use SMOTE previously to oversampling the training data. However, this time I think I need to use a leave-on group out (LOGO) cross-...
3
votes
0answers
431 views

Convolutional neural network with cross validation in Keras

I want to use K-fold cross-validation on my dataset of images. I am reading the data (images) from a directory. How do I use cross validation with convolutional neural network in Keras?
3
votes
1answer
61 views

Restrictions on my skewed validation data

I have a severely skewed data sets consisting of 20 something classes where the smallest class contains on the order of 1000 samples and the largest several millions. Regarding the validation data, ...
3
votes
0answers
31 views

Intuitive interpretation of ratios between training set scores and validation set scores

I'm training models with the usual setup where you hold back a portion (in my case, 20%) of the data just to see how your trained model generalizes to unseen data, to see if it's overfitting. When ...
3
votes
0answers
1k views

How to represent ROC curve when using Cross-Validation

I am performing k-Fold Cross Validation using a Logistic Regression classifier on a dataset and computing the ROC curve and the AUC for each fold. My desired output is one ROC curve with a ...
2
votes
1answer
35 views

Using cross validation score to perform feature selection

So to perform my feature selection I ran cross validation over and over again, each time trying different subsets of my attributes and repeated this until I got the best cross validation score I could ...
2
votes
1answer
27 views

leave one pair out cross validation

I am trying to train and validate my datasets which contains 17 datasets. I have divided them as 15 for training and 2 for validation. In the process, I train on 15 datasets and use the generated ...
2
votes
0answers
65 views

Cross-Validation for Unsupervised Anomaly Detection with Isolation Forest

I am wondering whether I can perform any kind of Cross-Validation or GridSearchCV for unsupervised learning. The thing is that I have the ground truth labels (but since it is unsupervised I just drop ...
2
votes
0answers
102 views

Cross validation for Collaborative filter-based recommendation systems

I am an absolute beginner and am trying to implement collaborative filter for furniture ecommerce (think wayfair). I need some guidance about cross-validation strategy. Situation: I am working on a ...
2
votes
0answers
156 views

Why does cross validation have a pessimistic bias?

My course notes list two reasons why cross-validation has a pessimistic bias. The first one is that the accuracy is measured for models that are trained on less data, which I understand. However, the ...
2
votes
1answer
418 views

Why does my CNN validation loss increase immediately, even with lots of data?

The Issue I've been working on a regression CNN implementation to predict time series data and have run into an issue where my validation loss and training loss diverge immediately during training, as ...
2
votes
1answer
62 views

Relation between Cross Validation and Confidence Intervals

I've read from a source which I forgot where that 'In cross validation, the model with best scores at 95% confidence interval is picked'. But according to my stat knowledge, in order for CI (...
2
votes
0answers
34 views

How to use a single GPU (vs. CPUs) in Tensorflow for forward inferencing (validation) vs. only for training

As mentioned in this question, it could be useful to harness a GPU vs. CPU(s) for validation. For example, in cross-validation, where the number of validation examples can exceed the number of ...
2
votes
1answer
183 views

Combine RepeatedStratifiedKFold and crossval

Is it okay to combine RepeatedStratifiedKFold and cross_val_score? The result in the example below are 30 accuracy values (3x10 splits). How to calculate the final accuracy value for the 3 times 10 ...
2
votes
0answers
86 views

R train(method="naive_bayes") and naiveBayes() very different performance

I am an R novice and having some difficulty. I was hoping R would be a good (flexible, easy) way to do machine learning of textual data. A few years ago, I wrote a naive Bayesian classifier (from ...
2
votes
2answers
67 views

Does high accuracy metrics with small (but equally sampled) dataset means a good model?

I have been training my CNN with 200 images per class for a classification problem. There problem is a binary classification one. And with the amount of test data ( 25 per class) I am getting good ...
2
votes
0answers
66 views

Performance diagnostics in mxnet gluon (e.g. plotting training vs validation loss over time)?

Tensorflow has tensorboard, is there any recommended way to plot classification error/loss over time in mxnet?
2
votes
1answer
2k views

K-fold cross validation of scikit-learn with confusion matrix of Keras

I intend to display a confusion matrix using Keras while K-fold of scikit-learn. My code using Keras is: ...
2
votes
1answer
760 views

How do I perform Leave One Out Cross Validation For Top n Recommendation Sytems?

I am new in making recommendation systems . I am using the surpriselib library to evaluate my recommendations. All the Accuracy Metrics are well supported in this library. But I also want to compute ...
2
votes
0answers
48 views

Is it possible to get the mean coefficent of regression after "Test & Score" in Orange using cross-validation?

The cross-validation devides the data into n folds and measures the accuracy n times therefore. The displayed RMSE in the Test & Score tool is a mean of all n runs, I think. But is it possible to ...
2
votes
1answer
600 views

Optimizing decision threshold on model with oversampled/imbalanced data

I'm working on developing a model with a highly imbalanced dataset (0.7% Minority class). To remedy the imbalance, I was going to oversample using algorithms from imbalanced-learn library. I had a ...
1
vote
0answers
20 views

Interpreting XGBoost's results (when they span between [0,0.5])

I would like to classify sentences into one of two different categories. I trained a XGBoost model over a search grid with k-folds cross-validation. My data represents sentences, and features ...
1
vote
1answer
17 views

Is there any better approach then K folds and nested K folds?

I am trying to understand what problem is K-folds solving. It does not seem to be solving data leakage at all, as we are still testing on test data and then taking an average of all test folds and ...
1
vote
1answer
46 views

Python scikit learn KFold function uneven train, test split

i have the following code below where i have noticed that the length of the train, test split from Kfold.split() is different for the last fold. Any reason why this may be happening and how i can go ...
1
vote
0answers
44 views

Cross Validation in Neural Networks

I am training a neural network and doing 10-fold cross validation to measure performance. I have read lots of documentation and forums telling that the set of weights that should be saved or ...
1
vote
0answers
18 views

How should you preprocess the data before K-fold validation?

I often see Kaggle notebook authors preprocessing the entire training data prior to splitting it for K-fold validation, but does this have a risk of leaking information into the validation set each ...
1
vote
0answers
27 views

How to implement kfold and cv into Hybrid feature selection and evaluate the classification model performance?

I have been working on a Hybrid feature selection combined with hyperopt package for hyperparameter tuning and I am thinking about evaluating the performance of several model classifiers. I looked ...
1
vote
1answer
61 views

Splitting the dataset manually for k-Fold Cross-Validation

I manually divided the dataset into three sets: train, test, and validation. Each set includes several folders, one for each patient. Each patient has many images from a different point of view. As a ...
1
vote
0answers
25 views

Contradictory learning curves in cross validation

I'm fitting gradient boosted decision trees (lightgbm) to model a regression problem. The data is extremely noisy, $R^2 \approx 0$. I'm trying to improve the fitting procedure using 10 fold cross ...
1
vote
0answers
74 views

How to pass manually split data to cross-validation

I have to perform a binary classification. My dataset is quite small 280 samples and quite imbalanced (1:10 ratio). I kept around 100 sample as testing and about 140 for training. My input variables ...
1
vote
0answers
53 views

Imbalanced dataset, finding the statistical significance of a Matthews Correlation Coefficient (MCC) in binary classification (what is a good MCC)?

I have a very imbalanced dataset. Thus, I am using MCC to evaluate the performance of various ML algorithms. It appears that literature is entirely lacking in ways to evaluate how good an MCC score is....
1
vote
2answers
18 views

Does adding a model complexity penalty to the loss function allow you to skip cross-validation?

It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of ...
1
vote
1answer
25 views

Can I apply different hyper-parameters for different sliding time windows?

Question Can I apply different hyper-parameters for different training sets? I can see the point of using the shared parameters but I cannot see the point of using shared hyper-parameters. The ...
1
vote
0answers
60 views

Train/validation/test and cross-validation on panel dataset

(Cross-posting a previous question from CrossValidated in case it is more suitable here: Train/Validation/Test and Cross-Validation on Panel Dataset ) I have a panel dataset, indexed by $Year$ and $...
1
vote
0answers
37 views

Unexpectedly long computation times with nested cross validation

Hello StackExchange community, I am trying to apply Nested Cross Validation on a pipeline to get a reliable estimate of the generalization error of my model. The pipeline includes two steps: Scaling ...
1
vote
0answers
102 views

Stratified K Fold Cross Validation in Orange: python script

I am using Orange to predict customer churn and compare different learners based on accuracy, F1, etc. As my problem is unbalanced (10% churn - 90% not churn), I want to oversample. However, when ...
1
vote
0answers
24 views

I'm worried that I'm training my model wrong

So I'm trying to classify some fashion mnist like photos into either boots or sneakers. I'm using a perception from sklearn to do so. The data set is a CSV containing pixel values. The model is ...
1
vote
0answers
108 views

Adaptive Resampling in Caret with Pre-specified Validation Set

I was wondering if this is the correct way to get adaptive sampling in caret working with a pre-specified validation set using index. I can get this to work using the 'cv' method in caret like so <...
1
vote
0answers
28 views

How can I do nested cross validation for multivariate Time series forecsting

I'm trying to do nested CV for my multivariate time series but I'm really confused how to do it. I have 7 Time series which are the inputs of my CNN model and one time series as target.Always when I ...
1
vote
1answer
162 views

Cross validation schema for imbalanced dataset

Based on a previous post, I understand the need to ensure that the validation folds during the CV process have the same imbalanced distribution as the original dataset when training a binary ...
1
vote
0answers
223 views

Hyperparameter tuning one-class svm

I have a problem where I am trying to apply a one-class svm to detect outliers. I am training on a dataset of true cases using a one-class radial svm and then predicting for both false and true cases. ...
1
vote
0answers
18 views

Does multiple cross validations after a model selection make sense?

Let be $M$ a model. Let also be $H_{1 \leq i \leq n}$ some hypothesis of $M$. I have a dataset $\mathcal{D}$ and I want to run $K$-fold cross validation on $\mathcal{D}$ to pick the best model $H_j = ...
1
vote
0answers
57 views

Is the forward chaining CV really suitable for time series?

If the time series distribution is non-stationary, we have to retrain our model once in a while (because it must forget the old dependencies and learn new ones). In forward chaining CV we use longer ...
1
vote
0answers
69 views

None of the known overfitting prevention techniques works for me, according to learning curves

I am working on HTRU2 dataset to evaluate classification models. Even though I obtain good results in terms of accuracy-MSE: I have an overfitting problem according to the learning curves below. In ...