Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Operation of the k-fold cross-validation method when using caret package

I have a question regarding the operation of the k-fold cross-validation method when using the caret package. As I understand it,...
1
vote
0answers
12 views

Is the forward chaining CV really suitable for time series?

If the time series distribution is non-stationary, we have to retrain our model once in a while (because it must forget the old dependencies and learn new ones). In forward chaining CV we use longer ...
0
votes
0answers
15 views

Use of variance of estimators in cross-validation [migrated]

Let's suppose we are using K-fold cross validation on a set of data of dimension $N_{data}$. We do not want to fix any parameter but just to get a confidence of the predictive power using the ...
0
votes
0answers
49 views

None of the known overfitting prevention techniques works for me, according to learning curves

I am working on HTRU2 dataset to evaluate classification models. Even though I obtain good results in terms of accuracy-MSE: I have an overfitting problem according to the learning curves below. In ...
0
votes
0answers
19 views

How to improve cross validation score? [closed]

I am working on regression problem with around 600 observations , train / test split = 0.2 and CV= 5, I have tired all the possible techniques to improve cross validation score : Removed outliers ...
1
vote
1answer
13 views

cross validation on whole data set or training data?

I am always having cross validation score smaller then the training score and I am performing cross validation on just training data is that normal thing ? Kfold = 5
1
vote
1answer
51 views

Interpretation for test score , training score and validation score in machine learning?

Interpretation for test score , training score and validation score ? what they actually tell us? What's an acceptable difference between cross test score , validation score and test score? If ...
0
votes
0answers
12 views

Term for Training and Validation Data combined

TL;DR Is there a commonly used term for the union of Training and Validation Data? The Problem It's sometimes hard to find and agree on the right name or term for the concept you're trying to ...
1
vote
1answer
29 views

Decision Trees change result at every run, how can I trust of my results?

Given a database, I split the data in train and test. I want to use a decision-tree classifier (sklearn) for a binary classification problem. Considering I already found the best parameters for my ...
0
votes
1answer
22 views

80-20 or 80-10-10 for training machine learning models?

I have a very basic question. 1) When is it recommended to hold part of the data for validation and when is it unnecessary? For example, when can we say it is better to have 80% training, 10% ...
0
votes
1answer
13 views

Output of multi-fold cross validation

I have been reading a lot about K-fold cross validation. By a way I attended a class project presentation on this topic recently and I still wonder if by the end of this validation method, one has K ...
1
vote
0answers
39 views

Understanding the output of the Random Forest method for classification

I'm using a Random Forest method to predict the behavior of failures at Period_12. My dataset has information about the eleven periods before, considering 112 subperiods (rows). Each one of these ...
0
votes
0answers
14 views

How to use Predefined Split for Randomized SearchCV

I'm trying to regularize my random forest regressor with RandomizedSearchCV. With RandomizedSearchCV the train and test are not ...
0
votes
0answers
16 views

Drastic drop in Somers' D ? Why?

I came across to find the correlation between the ratings assigned by two coaches to a same group of 40 players. I have tabulated the results as below: The Somers' D is 50%. However, for the case ...
0
votes
0answers
11 views

Is there a statistical law that can determine quantity of experiment and predict accuracy of results in my experimental context? [migrated]

I designed a biological protocol A which measures a continuous variable qA for each sample I designed a biological protocol B which measures a continuous variable qB for each same sample as above ...
1
vote
1answer
28 views

Why there is very large difference between cross validation scores?

I have a very simple regression model and I am doing the cross validation. When cv=10 the highest score i got is 60.3 and lowest is -9.7 which is useless. Average will be 30. No of row data set= ...
1
vote
1answer
52 views

Recommendations for statistical models given my dataset

I'm having a problem when using the k-fold cross-validation with the Random Forest method. One of the outputs is the error "Error in randomForest.default(x, y, mtry = param$mtry, ...) : Need at least ...
1
vote
1answer
23 views

Asynchronous Hyperparameter Optimization - Dependency between iterations

When using Asynchronous Hyperparameter Optimization packages such as scikit optimize or hyperopt with cross validation (e.g., cv = 2 or 4) and setting the number of iteration to N (e.g., N=100), ...
0
votes
1answer
39 views

Should bias updates be porportional to overfitting?

According to questions on the internet, the bias is a learnable parameter, and there are different solutions to updating it, but I failed to find a concise methodology of correctly updating biases ...
0
votes
0answers
18 views

How to validate logistic regression in SAS university edition

I am very new to SAS and I'm using the University Edition. I was provided with a pre-made training and testing set. I fit a binary logistic regression model with the training set, but now I want to ...
0
votes
0answers
8 views

Is test data needed on model with no hyper-parameter

I am doing classification using Linear Discriminant Analysis (LDA), which has no hyper-parameters. I am aware the difference between validation and test set, i.e. validation is used for hyper-...
4
votes
1answer
41 views

How fbprophet cross validation works

I am facing some issues to understand how cross_validation function works in fbprophet packages. I have a time series of 68 days (only business days) grouped by 15min and a certain metric : 00:00 ...
0
votes
1answer
21 views

Getting the leave-one-out error on least square regression to fit polynomials

I need to implement least square regression to fit polynomials of degree 1-27. I then need to get the leave-one-out error (kfold cross validation where k = n). After doing a lot of research it seems ...
3
votes
2answers
55 views

What is done first, cross validation or grid search?

When I have the data set to train a model with SVM, which procedure is performed first, cross validation or grid search? I have read this in a couple of books but I don't know in what order all this ...
1
vote
1answer
43 views

How to transform predicted results when doing cross-validation in sklearn?

I want to do cross-validation in sklearn like below, but the predicted result of X still need to be transformed to reduce the distance from y. How to do that with adding a custom function? ...
4
votes
1answer
25 views

Does k fold cross validation become less useful when number of observations is very large?

As seen in the accepted answer for variance of k-fold cross validation , the simulation shows that k-fold CV has the same test error rate for different values of k when n=200. Does this mean that k-...
0
votes
1answer
20 views

Different results obtained for OneVsOneClassifier (or OneVsRestClassifier) when using ordinary KFold and StratifiedKFold cross validation

When I fitted a OneVsOneClassifier (or OneVsRestClassifier), I noticed I obtained different results when I used ordinary KFold and StratifiedKFold cross validation. The testing set performance is much ...
3
votes
2answers
53 views

Shuffle the data before splitting into folds

I am running a 4-folds cross validation hyperparameter tuning using sklearn's 'cross_validate' and 'KFold' functions. Assuming that my training dataset is already shuffled, then should I for each ...
4
votes
2answers
262 views

Cross Validation - Why does more folds increase variation?

Can someone explain why increasing the number of folds in a cross validation increases the variation (or the standard deviation) of the scores in each fold. I've logged the data below. I'm working on ...
2
votes
1answer
22 views

How to select the best model from validation/training/holdout accuracy score

I have made my own function to log all the attempts at hyperparamter tuning, the following information is gathered from a 10 fold cross validation. But I am struggling to work out which model is best....
1
vote
1answer
13 views

When/how should I use the validation set for hyper-parameter sweeps for neural networks?

I know similar questions have been asked so many times, but I couldn't find the answer to this one particularly, at least not in a way that satisfied me. I am very confused about how to use ...
2
votes
0answers
16 views

Logic Check: Building a SKLearn Pipeline

I am new to the concept of building a pipeline in SKLearn and would appreciate some sense-checking to ensure that I am not leaking info from my training sets into my test set. Background: I have a ...
1
vote
2answers
25 views

Python and GridSearchCV how to eliminate input contains NaN error when using cross validation and decision tree classifier?

I am trying to do cross validation on Decision tree classifier for kaggle's titanic dataset. The first step after cleaning data is to split into train and test sets: ...
2
votes
1answer
39 views

Random Forest workflow?

I have a data-set comprised of a fairly large number of columns (over 1000) relative to the number of rows (370) that I am currently running a random forest regression on. I am a little confused with ...
2
votes
1answer
28 views

Overfitting with sklearn pipeline - reasons why?

So.... I've been playing around with this for FAR TOOO LONG now and I really need some advice. Most people on kaggle concat training and testing set TOGETHER and then pre scale the data, this seems ...
1
vote
2answers
16 views

is there cross validation for xgb classification for multi labels?

is there cross validation for xgb classification for multi labels? I have been search but can not find any cross validation for xgb classifier is using cross validation for xgb or xgb classifier ...
0
votes
1answer
15 views

Increasing samples increases variance

I've been running kfold cross validation with 10 folds and comparing it against a test set. Logging the score and the stdev along the way. Once I wad happy with my model I then run the estimator ...
0
votes
0answers
27 views

If validation accuracy is not improving what is wrong?

I am trying to do the classification for image %%time criterion = nn.CrossEntropyLoss() ...
1
vote
1answer
21 views

Using standard deviation as a metric for model selection

I'm really getting stuck with overfitting and I'm trying all I can to reduce it. I want't to write a metric to help score models in a cv loop. I'm using 10x5 folds and still getting out of sample ...
1
vote
1answer
40 views

is final fit with X,y or X_train , y_train?

I split the dataset with X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) and the fit ...
0
votes
0answers
30 views

How can I test my trained model on a completely new dataset? [duplicate]

Preface I have an annotated text dataset on hate speech. Simply put, the dataset consists of a column called text which includes a piece of text, and a column ...
4
votes
1answer
152 views

ROC AUC score is much less than average cross validation score

Using Lending club Dataset to find the propability of default. I am using hyperopt library to fine tune hyper parameter for an XGBclassifier and trying to maximize the ROC AUC score. I am also using ...
2
votes
1answer
66 views

Does it make sense to use train_test_split and cross-validation when using GridSearchCV to play with hyperparameters?

I was wondering if my methodology makes sense. I am using GridSearchCV with cross-validation to train and tune model hyperparameters for a bunch of different model types (e.g. Regression Trees, Ridge, ...
0
votes
0answers
11 views

SuperLearner Cross validation with iid time series

I created a number of ML models in R and I aim at combining them to form an ensemble. I learned about SuperLearner library which cross validates many models and returns the weight to each model in ...
1
vote
0answers
28 views

Stratified K fold same index present in both test and valid set

I am trying to do a stratified k fold cross validation for my dataset and want to keep an isolated 10% test set from the dataset and use the remaining for training and validation. Below is the code ...
1
vote
1answer
19 views

Dealing with issues in “test” predictons for single “items” (null values, standardization in place, etc)

I know this is kind of a broad question but I have tried to scour both this forum and the internet in general to no avail for this particular situation. So imagine I have a model trained for which, ...
1
vote
0answers
21 views

True test data when using leave-one-out cross-validation

I'm new to machine learning and I'm trying to use it for a personal project in R. I'm trying to use leave-one-out cross-validation for training my data. In particular, I'm running the following code: ...
1
vote
1answer
18 views

Is there a link between Training, Test errors based on k fold CV and not doing CV?

I am using Matlab to train a feedforward NN using Cross validation (CV) approach. My understanding of CV approach is the following. (Please correct me where wrong) Let ...
1
vote
1answer
32 views

Combine RepeatedStratifiedKFold and crossval

Is it okay to combine RepeatedStratifiedKFold and cross_val_score? The result in the example below are 30 accuracy values (3x10 splits). How to calculate the final accuracy value for the 3 times 10 ...
1
vote
0answers
19 views

Boosting, cross validation on small dataset

I have a really small dataset of 20 numbers and 20 potential predictors. (3 are important, 7 half important, 10 may be important) I use boosting (R-package mboost) to find the best predictive model. ...

1
2 3 4 5
8