Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

Filter by
Sorted by
Tagged with
4
votes
0answers
54 views

ground truth fit is worse than cross validated fit on noisy data?

I am having these weird results when playing around with cross validation that I would greatly appreciate to have any comments. Briefly, I have a lower mean squared error (MSE) when doing regression (...
0
votes
1answer
17 views

How to analyse the results of cross-validation do determine overfitting

I performed k-fold CV and measured the resulting average error (RMSE) for each fold. This was done with 5 folds, and 4 of the measurements gave similar errors (between 10% and 12%), but one of the ...
3
votes
2answers
40 views

When using Scikit Learn Grid Search, why are my train and cv scores high, but my test score is a lot lower?

I'm using scikit learn to run some models, and am very confused as to why my test score is so much lower than my cv score and my train score. At the start, I do a 80-20 train-test split. On the train ...
2
votes
0answers
21 views

Relation between Cross Validation and Confidence Intervals

I've read from a source which I forgot where that 'In cross validation, the model with best scores at 95% confidence interval is picked'. But according to my stat knowledge, in order for CI (...
1
vote
3answers
40 views

Should you turn off label smoothing when validating?

As the subject says. On one hand, the answer should be yes because label smoothing is a regularization feature and how can you know if it improves performance without turning it off? On the other hand,...
0
votes
1answer
22 views

plot gridsearch csv results how?

how can i plot my results from gridsearch csv? ...
0
votes
2answers
32 views

Validation Curve Interpretation

I have been reading about the validation_curve function from scikit learn. When I run this it takes too long. Therefore, I am plotting the results from grid search ...
0
votes
1answer
86 views
0
votes
1answer
35 views

Cross validation schema for imbalanced dataset

Based on a previous post, I understand the need to ensure that the validation folds during the CV process have the same imbalanced distribution as the original dataset when training a binary ...
0
votes
0answers
8 views

Neural network - calibration strategy & cross-validation

I have a hard time articulating the different parts of a calibration process of a relatively vanilla neural network. I am mostly concerned with : Grid search for the regularisation hyperparameter ...
5
votes
2answers
93 views

Why is 10 considered the default value for k-fold cross-validation?

I understand very well what k-fold cross-validation is. In my studies, and at work, I've always heard something along the lines of: We most often use k=10 ...
0
votes
0answers
27 views

XGBOOST/lLightgbm over-fitting despite no indication in cross-validation test scores?

We currently work on a project where we aim to identify a set of predictors that may influence the risk of a relatively rare outcome. We are using a semi-large clinical dataset, with data on nearly ...
1
vote
2answers
45 views

Main options on how to deal with imbalanced data

As far as I can tell, broadly speaking, there are three ways of dealing with binary imbalanced datasets: Option 1: Create k-fold Cross-Validation samples randomly (or even better create k-fold ...
1
vote
1answer
37 views

Training accuracy is ~97% but validation accuracy is stuck at ~40%. What does it imply? [duplicate]

Training accuracy is ~97% but validation accuracy is stuck at ~40%. I can not understand the meaning of two concepts and their relationship.
0
votes
0answers
17 views

Consecutive Feature Selection-CV and Model Selection-CV

I want to ask a question about general workflow of algorithm development. I want to include a "feature selection with Random Forest" step into my workflow but I have doubts about data leakage. It is ...
-1
votes
1answer
29 views

Why when my local cv of loss decreases, my leaderboard's loss increases?

I got a cv log_loss of 0.3025410331400577 when using 4-fold cross-validation and my leaderboard (with 30% of test dataset) got 0.26514. I further did feature engineering and added some features to the ...
0
votes
0answers
19 views

cross validation with early stopping

I am evaluating the performance of my deep learning model. My model is using an early stopping technic and I would like to do 5-fold cross-validation. Would below be the correct performance evaluation?...
4
votes
2answers
60 views

Robustness of hyperparameter tuning

I use a Bayesian hyperparameter (HP) optimization approach (BOHB) to tune a deep learning model. However, the resulting model is not robust when repeatedly applied to the same data. I know, I could ...
1
vote
0answers
63 views

Hyperparameter tuning one-class svm

I have a problem where I am trying to apply a one-class svm to detect outliers. I am training on a dataset of true cases using a one-class radial svm and then predicting for both false and true cases. ...
1
vote
3answers
35 views

Does shuffling data for time series forecasting help?

So I am trying time series forecasting using LSTM's. The aim is to predict $Y$ given $X$ using regression. I had already converted the input data into a sliding window format such that if my input ...
6
votes
2answers
376 views

What is the proper way to use early stopping with cross-validation?

I am not sure what is the proper way to use early stopping with cross-validation for a gradient boosting algorithm. For a simple train/valid split, we can use the valid dataset as the evaluation ...
0
votes
2answers
28 views

Leave - one - out - Cross Validation KNN R

I have a dataset and I divided it into test data and train data. Can anyone suggest how to perform LOOCV for KNN regression? Is there any library? ...
0
votes
0answers
20 views

Find SVR custom kernel weights automatically

I am building an SVR based predictor using a custom kernel. My kernel looks at two features and I want to give different weights to those features. Example: ...
7
votes
3answers
152 views

What needs to be done to make n_jobs work properly on sklearn? in particular on ElasticNetCV?

The constructor of sklearn.linear_model.ElasticNetCV takesn_jobs as an argument. Quoting the documentation here n_jobs: int, ...
2
votes
2answers
70 views

What is the purpose of a confusion matrix in a classification problem?

I am studying machine learning. After some research I understood that a typical workflow for a classification problem (after having prepared the data) is the following: Split data in test, train and ...
1
vote
2answers
23 views

Training, cross validation and testing accuracy (RMSE and R2) differs when using different shuffles and splits

I have a very small data set of 60 observations. My training, cross-validation and testing accuracy (RMSE and R-squared) differ in a considerable amount when using different random states while ...
1
vote
2answers
26 views

Is this over-fitting or something else?

I recently put together an entry for the House Prices Kaggle competition for beginners. I decided to try my hand at understanding and using XGBoost. I split Kaggle's 'training' data into 'training' ...
-1
votes
1answer
23 views

Proper workflow for model selection and hyperparameter tuning using cross validation

I have been trying to teach myself about machine learning and wanted to make sure I had the right idea about model selection, hyperparameter tuning, and cross validation. So given a data set, my ...
0
votes
1answer
27 views

How to approach an extremely unbalanced time series dataset

I need to classify a relatively small time series dataset. Training set dimensions are 5087 rows (to classify) by 3197 columns (time samples) which are (or should be as far as I understood) the ...
0
votes
1answer
57 views

How to train-test split and cross validate in Surprise?

I wrote the following code below which works: ...
0
votes
1answer
31 views

plotting a decision tree based on gridsearchcv

i was trying to plot the decision tree which is formed with GridSearchCV, but its giving me an Attribute error. ...
0
votes
0answers
12 views

Improve performance for known unlabelled test set

I'm training a machine learning model from a training set of $1000$ samples (but around $23000$ features). I'm fairly content with my cross-validated results, but would like to improve it further by ...
0
votes
1answer
43 views

(De-)Scaling/normalizing input and output data inside Keras model as layer

I am building a 2-hidden layer MLP using Keras. I'm using a SciKit learn wrapper to be able to use the GridSearchCV functionality. My sample-size is limited, ...
0
votes
0answers
95 views
0
votes
1answer
32 views

Shouldn't training and validation loss be approximate before the first epoch is finished?

I'm having this burning question in my head, and I couldn't find the answer anywhere. During training, at least in Keras, the training loss is computed on the current batch, so the weights can be ...
0
votes
0answers
11 views

Given a dataset, how to test your model against the test set if you used StratifiedKFold and standardized train and validation sets per fold?

I couldn't find an answer to the following issue and I kinda feel stuck... I have a dataset and I want to split it as follows: 90% for train and validation 10% for test Now, I want to use ...
0
votes
1answer
60 views

choosing model based on last or best iteration on validation set

This is a very basic question, however I haven't found a satisfying answer until now. When training a neural network we must choose the number of epochs. The usual advice is to train as long as the ...
2
votes
1answer
44 views

Custom functions and pipelines

I'm not really used to working with pipelines, so I'm wondering how can I use custom functions and pipelines. Situation: I want to fill some missing values with the mean but using groups based on ...
0
votes
0answers
16 views

How to use a single GPU (vs. CPUs) in Tensorflow for forward inferencing (validation) vs. only for training

As mentioned in this question, it could be useful to harness a GPU vs. CPU(s) for validation. For example, in cross-validation, where the number of validation examples can exceed the number of ...
1
vote
0answers
14 views

Does multiple cross validations after a model selection make sense?

Let be $M$ a model. Let also be $H_{1 \leq i \leq n}$ some hypothesis of $M$. I have a dataset $\mathcal{D}$ and I want to run $K$-fold cross validation on $\mathcal{D}$ to pick the best model $H_j = ...
1
vote
0answers
31 views

Is the forward chaining CV really suitable for time series?

If the time series distribution is non-stationary, we have to retrain our model once in a while (because it must forget the old dependencies and learn new ones). In forward chaining CV we use longer ...
0
votes
0answers
53 views

None of the known overfitting prevention techniques works for me, according to learning curves

I am working on HTRU2 dataset to evaluate classification models. Even though I obtain good results in terms of accuracy-MSE: I have an overfitting problem according to the learning curves below. In ...
1
vote
1answer
19 views

cross validation on whole data set or training data?

I am always having cross validation score smaller then the training score and I am performing cross validation on just training data is that normal thing ? Kfold = 5
1
vote
1answer
80 views

Interpretation for test score , training score and validation score in machine learning?

Interpretation for test score , training score and validation score ? what they actually tell us? What's an acceptable difference between cross test score , validation score and test score? If ...
0
votes
0answers
17 views

Term for Training and Validation Data combined

TL;DR Is there a commonly used term for the union of Training and Validation Data? The Problem It's sometimes hard to find and agree on the right name or term for the concept you're trying to ...
1
vote
1answer
43 views

Decision Trees change result at every run, how can I trust of my results?

Given a database, I split the data in train and test. I want to use a decision-tree classifier (sklearn) for a binary classification problem. Considering I already found the best parameters for my ...
0
votes
1answer
40 views

80-20 or 80-10-10 for training machine learning models?

I have a very basic question. 1) When is it recommended to hold part of the data for validation and when is it unnecessary? For example, when can we say it is better to have 80% training, 10% ...
0
votes
1answer
16 views

Output of multi-fold cross validation

I have been reading a lot about K-fold cross validation. By a way I attended a class project presentation on this topic recently and I still wonder if by the end of this validation method, one has K ...
1
vote
0answers
41 views

Understanding the output of the Random Forest method for classification

I'm using a Random Forest method to predict the behavior of failures at Period_12. My dataset has information about the eleven periods before, considering 112 subperiods (rows). Each one of these ...
0
votes
0answers
18 views

How to use Predefined Split for Randomized SearchCV

I'm trying to regularize my random forest regressor with RandomizedSearchCV. With RandomizedSearchCV the train and test are not ...

1
2 3 4 5
9