Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

Filter by
Sorted by
Tagged with
0
votes
0answers
15 views

Number of samples in cross validation? [closed]

How do I check the number of training sample and number of test samples when I have used cross validation(cv=10)?
0
votes
0answers
17 views

How to find out if someone cheated by training on the test set?

Do approaches exist to see if somebody (e.g. authors of a scientific publication) unfairly improved their results by using (parts of) the test set also for training? Any ideas? Using a new and ...
0
votes
0answers
44 views

Time series imputation benchmark

In a work, I have to benchmark different algorithms to fill in missing values in time series. I insist on the fact that this is imputation and not forecasting. In my case, I have access to 15 years of ...
1
vote
1answer
27 views

What is the best practice for tuning hyperparameters using validation data?

I'm building a binary classifier, using task-transfer from resnet and a total training set of 300 images. Initially I put aside 100 images as validation, and tuned the hyperparameters, each time ...
0
votes
0answers
26 views

Confusion regarding accuracy and individual class performance

Consider a three-class classification problem where avg_cm1 and avg_cm2 are two average confusion matrices across 3 folds from ...
0
votes
1answer
19 views

Does the model(best fitting line/curve) changes when the training data is changed in the cross validation?

From my understanding - a machine learning algorithm goes through the inputs (independent variables) and predicts the output (dependent variable). I believe, what line/curve would best define the ...
1
vote
0answers
10 views

Can I dynamically change the hyper-parameters of a model?

Question Can I apply different hyper-parameters for different training sets? I can see the point of using the shared parameters but I cannot see the point of using shared hyper-parameters. The ...
0
votes
0answers
14 views

Cross validation with GridSearchCV or train-val-test split

I have a question regarding the CV in GridSearchCV. To test my model should I split my data into 3: training, validation, test? For easy understanding let's say my data is split into training with 60% ...
1
vote
0answers
18 views

Train/validation/test and cross-validation on panel dataset

(Cross-posting a previous question from CrossValidated in case it is more suitable here: Train/Validation/Test and Cross-Validation on Panel Dataset ) I have a panel dataset, indexed by $Year$ and $...
0
votes
0answers
4 views

How to apply feature selection in cross validated bagging

Normally in cross validation decision tree, feature selection will occur with training data but in bagging ensemble the training data is bootstrapped. How can I apply feature selection in cross ...
0
votes
2answers
26 views

How can i solve the classification's problem with cross validation in LogisticRegression?

I want to make a data frame with most repeated word in sentences and make a classification via Logistic-Regression. I tried to write the steps clearly in codes. ...
0
votes
0answers
12 views

Orange Data Mining - Data Sampler Widget

In the Data Sampler widget, there is an option for Cross validation for which one fold can remain unused. Is this option for nested cross validation alternating which fold you hold back?
1
vote
1answer
24 views

Question about information leakage

I am well aware that to avoid information leakage, it is recommended to fit any transformation (e.g., standardization or imputation based on the median value) on the training dataset and applying it ...
1
vote
0answers
35 views

Unexpectedly long computation times with nested cross validation

Hello StackExchange community, I am trying to apply Nested Cross Validation on a pipeline to get a reliable estimate of the generalization error of my model. The pipeline includes two steps: Scaling ...
0
votes
1answer
27 views

When to use cross-validation?

Cross-validation Hi, I'm deploying machine learning models in my MSc thesis using Weka. I have noticed that when I use 10-fold cross-validation in the training dataset I get low evaluation metrics ...
3
votes
0answers
42 views

How to properly do feature selection when comparing different models?

Context: I'm currently crafting and comparing machine learning models to predict housing data. I have around 32000 data points, 42 features, and I'm predicting housing price. I'm comparing Random ...
0
votes
2answers
33 views

Kfold or cross_val_score

I am all new with ML. I try to understand what is Kfold and cross_val_score. I made this model: ...
0
votes
2answers
22 views

Cross-validated average: metrics mean or ensembling probabilities?

Let's say I have 5 models cross-validated via leave-one-out strategy. I have the predictions and scores of each model. Now, it's time to calculate the average for the set of 5 models - am I supposed ...
1
vote
0answers
12 views

How to cross validate WDBC.csv breast cancer classification dataset in Stacked Autoencoders? [closed]

I need kind guidance regarding the context of how to cross validate WDBC.csv (Wisconsin Breast cancer diagnostic) dataset for breast cancer binary classification in Stacked Autoencoders as I put the ...
0
votes
1answer
44 views

K-Fold cross validation and data leakage

I want to do K-Fold cross validation and also I want to do normalization or feature scaling for each fold. So let's say we have k folds. At each step we take one fold as validation set and the ...
1
vote
1answer
48 views

K-fold cross-validation with validation and test set

For a project I want to perform stratified 5-fold cross-validation, where for each fold the data is split into a test set (20%), validation set (20%) and training set (60%). I want the test sets and ...
1
vote
0answers
35 views

Stratified K Fold Cross Validation in Orange: python script

I am using Orange to predict customer churn and compare different learners based on accuracy, F1, etc. As my problem is unbalanced (10% churn - 90% not churn), I want to oversample. However, when ...
0
votes
0answers
30 views

Overfitting in imbalanced dataset

I am working on a dataset related to an insurance company and the objective is to predict if the insurance buyer will claim their travel insurance or not. Training data: https://raw.githubusercontent....
2
votes
1answer
58 views

Cross validation for unbalanced dataset using Orange data mining tool

I am using the Orange data mining tool to build and analyze models (decision tree, ANN, ...) predicting customer churn. As this is an imbalanced class problem (10% churn, 90% not churn), I need to ...
0
votes
1answer
17 views

SKLearn - Understanding Discrepancy Between LogisticRegressionCV classification_report and scores_

Cross-posting from Stack Overflow: I'm running into a weird situation where my sklearn LogisticRegressionCV model is apparently getting 100% accuracy (the lack of ...
0
votes
1answer
68 views

why should i do target encoding within cv loop?

i wish to use target encoding, using the category encoders sklearn library. I don't really understand why it is necessary to include this as a step in a sklearn pipeline WITHIN the cross validation ...
2
votes
1answer
35 views

What parameters to use when normalising training, validation, and testing data?

I know a similar post was made here, but I wanted to ask some follow up questions. I am conducting a cross-validation search to find values of a set of hyper-parameters and need to normalise the data. ...
0
votes
1answer
25 views

Compare cross validation values of Bernoulli NB and Multinomial NB

I'm testing the Multinomial NB and Bernoulli NB on my dataset and I'm using the cross validation score to better understand which of the two algorithms work better. This is the first classifier: ...
1
vote
2answers
53 views

How plot GridSearch results?

I trained an SVM model with GridSearch ...
1
vote
1answer
19 views

nested CV feature selection

I have a small dataset of 150 records with 25 features (too small to do train/test). I'm using nested cv for both hyperparameter tuning and feature selection. 10cv in the outer loop, 5 cv in the inner ...
0
votes
0answers
24 views

creating cv folds from a dataset that has to be grouped on two featues and also time series data

Edit Dec 3 2020: updated the code Edit: Following are the two changes 0a. uses sets to check there are duplicate random numbers generated 0b. We are now going from minority class to majority class to ...
-1
votes
1answer
19 views

Can the use of EarlyStopping() offset overfitting problems caused by validation_split?

Keras gives users the option, while fitting a model, to split the data into train/test samples using the parameter "validation_split. Example: ...
1
vote
1answer
43 views

small dataset CV

I have a very small dataset ( 150 records) with 20 features, trying to predict a binary outcome. Due to the small size, i chose to do 10 CV instead of train/test as the train/test split. I was ...
0
votes
0answers
17 views

Using accuracy metric during training for unbalanced multiclass classification

I am training a convolutional neural network and the sensitivity and precision of the minority class is what is most important to me. I am using 10-Fold cross validation, and the test fold is ...
1
vote
0answers
22 views

I'm worried that I'm training my model wrong

So I'm trying to classify some fashion mnist like photos into either boots or sneakers. I'm using a perception from sklearn to do so. The data set is a CSV containing pixel values. The model is ...
2
votes
1answer
38 views

When developing machine learning models, is the size of each class in the test set important?

I am thinking about the prospective application of a trained classifier in a real-world context. We know that when we do over/under-sampling to balance our dataset, we never touch the testing set as ...
0
votes
0answers
28 views

SVM overfitting with consistent validation results

I have some imbalanced (1400 samples of which 250 are +ve) data for a binary classification problem and I am running an SVM grid search optimising for precision. I am trying 3,4,5,6,7,and 8 stratified ...
0
votes
0answers
16 views

Preprocessing and feature selection in group k fold

I have experimental data collected from 10 people. From each person, 100 data points were collected under condition A, and 100 data points were collected under condition B. So, in total I have 10*(100+...
1
vote
1answer
29 views

How to understand Learning Curves

I've developed two models and wanted to test them. However, I don't know how to intepret the results of the learning curves properly. For Model 1: ...
1
vote
2answers
19 views

Is Cross Validation needed for regression if you already know the predictors in your model?

Let's say you want to model the behavior of Y = X1 + X2 and you know that this is the model you want to make. Whether or not that approximates the true relationship well is unknown. But since you ...
0
votes
1answer
39 views

Compare cross validation and test set results

I am having a hard time understanding the results of a cross validation test and a test run on a test set. First I made the following pipeline: ...
5
votes
1answer
84 views

Can one perform Feature Selection on a subset of training data?

I have a training data set with almost one million rows and I am considering eight features initially. My machine learning model will be Random Forest regressor. In Section 3.4.7 of "Feature ...
1
vote
1answer
49 views

nnet in caret. Bootstrapping or cross-validation?

I want to train shallow neural network with one hidden layer using nnet in caret. In trainControl, I used method = "cv" to perform 3-fold cross-validation. The snipped the code and results ...
0
votes
1answer
59 views

How to train with cross validation? and which f1 score to choose?

I got similar results in 2 models which consists of similar algorithms. Model 1 with cv=10 has a f1'micro' of 0.941. See code below. Model 2 only train test split (no cv) has f1'micro' 0.953. Now here ...
2
votes
0answers
68 views

Why does cross validation have a pessimistic bias?

My course notes list two reasons why cross-validation has a pessimistic bias. The first one is that the accuracy is measured for models that are trained on less data, which I understand. However, the ...
0
votes
1answer
44 views

Machine Learning validation data returns 100% accuracy [closed]

I'm Testing a Machine Learning model with validation data returns that return 100% correct answers, is it overfitting or the model works extremely well, do I need to continue training on more data? I'...
-1
votes
1answer
42 views

Model to choose with Cross Validation or not?

I made different tests on an imbalanced dataset and got these results: Model 1 = train test validation split + Cross Validation(cv=10) --> f1'micro' 0,95 Model 2 = train test split + smote method ...
2
votes
2answers
43 views

What is the next step after k fold CV?

I came across this video lecture https://www.youtube.com/watch?v=wjILv3-UGM8 on k fold cross validation (CV). The algorithm given in the video lecture is presented below: for k = 1:5 train on all ...
0
votes
1answer
28 views

Same confusion matrix when changing DecisionTreeClassifier parameters

I'm trying to build my first Decision Tree Classifier using the Iris dataset in the sklearn library. This is my first sample code: ...

1
2 3 4 5
10