Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

Filter by
Sorted by
Tagged with
0
votes
2answers
27 views

Accuracy of KFold Cross Validation for Neural Network

I have a neural network that Im evaluating using 10 -Fold cross validation. The validation accuracy for a fold changes alot during training in the range of -+10% So for example the validation ...
1
vote
2answers
22 views

Cross validation vs leave one out

I have found the following definitions, but I don't really see the difference. cross validation Method for testing classification and prediction models. The data are randomly split into N partitions (...
1
vote
2answers
26 views

Acceptable variation in accuracy of each k fold when using K-Fold Cross Validation?

I have a relatively small dataset consisting of 1432 samples. I have trained a Random Forest Classifier and performed KFold CV. The results of running 10 Fold CV are as follows: ...
1
vote
0answers
14 views
2
votes
1answer
32 views

Cross-Validation: Repeated K-Fold/Group K-Fold

Repeated K-Fold vs Group K-Fold As per my understanding from sklearn docs Repeated K-Fold: RepeatedKFold repeats K-Fold n times. It can be used when one requires to run KFold n times, producing ...
1
vote
0answers
26 views

How to avoid bias when optimizing time-series model?

I have 4000 days of data. I am trying create a time-series model with parameters P to forecast the value of the target Y using the last N days of data. The parameters P include: lookback window for ...
0
votes
2answers
74 views

how to use standardization / standardscaler() for train and test?

At the moment I perform the following: ...
1
vote
0answers
13 views

Is kFoldLoss simply the average classification error?

I'm using a kNN classifier in MATLAB to classify ECG signals, with 5 fold cross validation I get almost 97% accuracy. However, I'm not entirely sure if this is truly the accuracy value. By default the ...
0
votes
1answer
19 views

which scoring function for validation_curve (regression)?

Is there any thumb of rule which scoring function should be used for e.g. the validation_curve? Atm I try to study the difference between several optimizers: ...
0
votes
0answers
7 views

Differnt loglikelihood values in RandomSearch/GridSearch and LDA . How to synchronize them?

I am doing a sensitivity analysis for some lda parameters in python. This is what I want to do: -Do RandomSearch to find optimal Parameters and therefore the optimal lda model. As Validation measure ...
0
votes
0answers
10 views

Standardizing in each fold - Learning Curve

Problem Description Hello, I have a classification problem and I want to perform cross validation (with hyper parameter tuning) in order to evaluate the generalization of my models. Basically the ...
0
votes
0answers
7 views

Hypothesis Test for Classifier, but without the original model

I just trained a number of DenseNet-based CNN models using an optimization technique that I created for my master dissertation. The problem is, my advisor wants me to perform a hypothesis test to ...
0
votes
2answers
41 views

K fold cross validation reduces accuracy

I am working on a machine learning classifier and when I arrive at the moment of dividing my data into training set and test set Iwant to confron two different approches. In one approch I just split ...
0
votes
0answers
15 views

CNN models comparison

I coded a 38 layer CNN and 8 layer CNN but there's something wrong in my 38 layer CNN, which doesn't learn anything. Not able to fugure out what's wrong. They were trained on CIFAR.
0
votes
1answer
32 views

Correctly evaluate model with oversampling and cross-validation

I'm dealing with a classic case of dataset with binary imbalanced target (event 3%, non event 97%). My idea is to apply some sort of sampling (over/under, SMOTE etc.) to address the issue. As I see, ...
0
votes
0answers
10 views

Track validation_curve during hyperparameter optimization

To study the influence of a single (hyper-)parameter, I use validation_curve: ...
0
votes
2answers
28 views

Is using cross validation on your entire dataset acceptable when dealing with a small sample size

Normally my practice includes using k-folds cross validation on a subset of my dataset and keep a final test set. When dealing with an exceptionally small dataset, is using cross validation on the ...
0
votes
2answers
60 views

cross validation issues

I have come here from this great answer. I have come across many approaches for using cross validation and the answer to the attached question is by far explaining it the best to me. My dilemma is ...
0
votes
3answers
35 views

Cross-fold validation done on whole dataset or training set?

I have a dataset of 77 samples with 302 features with two labels (0,1). I trained an SVM with gridsearch (cv=5) to perform binary classification. In one run of my script, I do a test-train split, ...
0
votes
1answer
33 views

Difference between validation and prediction

As a follow-up to Validate via predict() or via fit()? I wonder about the difference between validation and prediction. To keep it simple, I will refer to train, <...
1
vote
1answer
50 views

Hyperparameter tuning and cross validation

I have some confusion about proper usage of cross-validation to tune hyperparameters and evaluate estimator performance and generalizeability. As I understand it, this would be the process you ...
0
votes
0answers
45 views

Value Error: MSLE & CrossVal

I'm trying to run cross validation with mean squared log error with sklearn and getting the following error message: ...
0
votes
1answer
76 views

clarification on train, test and val and how to use/implement it

So far I think I understood the differences between the training, test and validation set. Basically it is like in this image: Training set: The data where the model is trained on Validation set: ...
0
votes
2answers
25 views

Drawing validation set from test set

I am building a 3 neural network models on dataset that is already separated to train and test sets. From my analysis, I found that this dataset has values on test set which don't exist in the train ...
1
vote
1answer
38 views

K-fold-cross-validation if training dataset is much smaller than test dataset?

I'm a beginner in machine learning and I have a special case in which I have only a small training dataset of about 500 images and a test dataset of 10,000 images. Does it still make sense to do a 10-...
1
vote
1answer
17 views

How to construct validation set for time series for NN?

I would like to train my model with a validation set. As the data is a time series I have to use timeseriessplit: ...
2
votes
2answers
260 views

Advice and Ideas appreciated - Machine Learning one man project

I have a project where I am supposed to start from scratch and learn how machine Learning works. So far everything is working out better than expected but I feel as I am offered to many ways to choose ...
3
votes
2answers
157 views

XGBoost validation for number of trees

I have a simple Question: I am using XGBoost to classify some data: 1.) With 100 estimators I have following scores(roc_score): train_data : 98.5 validation_data : 97.2 2.) With 500 ...
0
votes
0answers
10 views

Recursive feature elimination on train data or complete dataset and dummy encoding

I am using RFE with logistic regression. I will also be doing cross validation with RFE (RFECV in sklearn) to get the optimum number of features. I am not sure whether to use RFECV on just train ...
0
votes
1answer
45 views

Cross-validation average score

I am using Repeated K-folds (RepeatedKFold(n_splits=10, n_repeats=10, random_state=999) from sklearn) to provide reliable scores for a linear regression on my ...
0
votes
0answers
7 views

Cross_validation is decreasing accuracy?

I have certain dataset to train a model. The dataset is not very small in size. First, I split the dataset into training and validation data using traintestsplit (80-20), train the model on training ...
3
votes
1answer
28 views

Cross validation Vs. Train Validate Test cont'd

Cross validation Vs. Train Validate Test I have one further question relating to @louic answer in the post above: "Training happens k times, each time leaving out a different part of the training ...
2
votes
3answers
55 views

Publish without validation score?

My mentor wants me to write and submit an academic paper reporting a predictive model, but without any validation score. Everything I have read in textbooks or the Internet says that this is wrong, ...
1
vote
1answer
34 views

Cross validation test and train errors

I came across this sort of flowchart: Below the flowchart, this is what appears: “Given a training set, cross-validation error is computed for each configuration of tuning parameters (λ,d). The ...
1
vote
0answers
38 views

Data leak when training on future data and testing on present [closed]

Given a time series dataset. Using simple train_test_split, then reversing the train to test and test to train i.e. using the future data to train and present data to test, where does one induce leak ...
1
vote
1answer
23 views

Validity of cross-validation for model performance estimation

When applying cross-validation for estimating the performance of a predictive model, the reported performance is usually the average performance over all the validation folds. As during this procedure,...
1
vote
1answer
30 views

What to do about non-responses to demographic survey questions?

Can anyone point me to some think pieces on what to do with census type data where at least some of the people surveyed do not self-identify a race/ethnicity, gender, or other demographic data? In ...
0
votes
1answer
18 views

Cross validation while preserving a column (not the target ) distribution

So i'm doing cross validation and then i'm predicting using all the data on a test set ( a hold-out set ). My hold-out set has the same ratio on a column than the train ( seems thats how the test set ...
0
votes
1answer
17 views

Interpreting a curve val_loss and loss in keras after training a model

I am having trouble understanding the curve val_loss and loss in keras after training my model. Can anyone help me understand ...
4
votes
1answer
109 views

sklearn's cross_validate does not work with catboost

I would like to use cross validation with catboost. Since I do not just want to use catboost but also sampling I am using a ...
1
vote
1answer
81 views

Oversampling only balances the training set, what about the testing set?

In a case of imbalanced data classification, I know that we only oversample the training set (to prevent data leakage from training to testing subsets), but what if there are no positive data points ...
2
votes
3answers
79 views

Should I perform cross validation only on the training set?

I am working with a dataset that I downloaded from Kaggle. The data set is already divided into two CSVs for Train and Test. I built a model using the training set because I imported the train CSV ...
2
votes
1answer
39 views

Significant drop from validation accuracy to test accuracy

I am more familiar with classification tasks, though I have been working on a regression problem. I was given a large training dataset (>70k samples) and an independently collected test set (~2k). I ...
0
votes
2answers
41 views

Does high accuracy metrics with small (but equally sampled) dataset means a good model?

I have been training my CNN with 200 images per class for a classification problem. There problem is a binary classification one. And with the amount of test data ( 25 per class) I am getting good ...
4
votes
1answer
44 views

help understanding nested cross validation

From what I read online, nested CV works as follows: I divide my whole data in k folds, where k-1 folds are the train set and one fold is the test set. There is the inner CV loop, where we may ...
1
vote
1answer
41 views

choosing classifiers

For what I read the 5x2cv t test is "a procedure for comparing the performance of two models (classifiers or regressors) that was proposed by Dietterich to address shortcomings in other methods such ...
0
votes
1answer
50 views

sckit-learn Cross validation and model retrain

I want to train a model and also perform cross validation in scikit-learn, If i want to access the model (For instance to see the parameter's selected and weights or to predict) i will need to fit it ...
3
votes
1answer
31 views

CrossValidation using glmnet and very high values of Lambda?

I am trying to run crossvalidation (folds=10) using glmnet library on my dataset. My outcome of interest is BMI and predictors include a set of clinical variables. My final goal is to use elastic-net ...
1
vote
1answer
35 views

Using of 100s of Binary features in regression model

I have 100s of columns with binary values [0, 1] plus some extra columns without binary values. I am trying to do regression model but the model performance is very low. For non-binary features, I ...
4
votes
1answer
157 views

Interpretability of RMSE and R squared scores on cross validation

I'm working on a regression problem with 30k rows in my dataset, decided to use XGBoost mainly to avoid processing data for a quick primitive model. And i noticed upon doing cross-validation that ...