Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

Filter by
Sorted by
Tagged with
1
vote
0answers
22 views

How to properly apply CrossValidation and/or split the dataset?

I have a kind of particular problem and do not really now how to properly apply CrossValidation in this scenario. There is one big data set with 100.000 samples, 99.000 y=0, 1.000 y=1 Each sample has ...
2
votes
1answer
23 views

Ridge regression model creation using grid-search and cross validation

I created python code for ridge regression.For that I used cross validation and grid-search technique in together. i got output result. I want check whether my regression model building steps correct ...
1
vote
1answer
16 views

Making sense of a accuracy plot for a 5 fold training using random forest

I'm using sklearn.model_selection.learning_curve for 5 fold training of data. The code is as given below. ...
2
votes
1answer
43 views

How to Bootstrap dataset for 10000 AUC scores?

I am new to ML and trying to learn the nuances. I work on a binary classification problem with 5K records. Label 1 is 1554 and Label 0 is 3554. What I currently do is 1) split the data into train(...
0
votes
1answer
22 views

splitting into train test by train_test_split of float values?

How to split into train test by train_test_split of float values ? I used LabelEncoder but I have about 300K lines and when I used the cross_val I saw ...
1
vote
0answers
24 views

Finding optimal threshold in extreme value theory - Peak Over Threshold-method (POT)

I am using r and doing a project with extreme values. Now I need to find optimal threshold based on charts outputs but I can't find any information how to interprete them. I am using ...
1
vote
1answer
21 views
0
votes
0answers
24 views

TypeError: 'KFold' object is not callable

I am trying to perform K-Fold Cross Validation with Scikit Learn: from sklearn.model_selection import KFold KFold but I am getting ...
1
vote
3answers
24 views

Will the MAE of testing data always be higher than MAE of training data?

On the Kaggle Course Page the chart below shows that MAE of testing data is always higher than MAE of training data. Why is this the case? Is it only limited to DecisionTreeRegressor model? Or the ...
1
vote
2answers
32 views

How to get sensitivity and specificity for multi-class classification for each fold of cross validation?

I am working on a multi-class classification consisting of 4 classes. I am applying 5-fold cross-validation on it and would like to get the sensitivity (recall) and specificity score for each of those ...
3
votes
2answers
27 views

What is meaning of zip(kfold.split(X, Y) in sklearn

What is meaning of zip(kfold.split(X, Y) in sklearn? for (train, test)in zip(kfold.split(X, Y)):
1
vote
2answers
43 views

K-Fold Cross Validation for NNs

When using K-Fold CV, is it still useful to have a Train/Validation/Test split? Or simply just a Train/Test? I.e. split up data into k bins, and leave one out for testing, train on the rest, and take ...
1
vote
1answer
11 views

Truncating float/doubles for reproducibility

I deploy machine learning models (typically GPU) to a variety of environments. I work sort of at the edge of ML R&D and devops, so I am really big into reproducibility, and one thing that drives ...
2
votes
0answers
29 views

how do we perform time-series walk forward analysis

I recently came across the technique called as walk forward time-series analysis which is equivalent to cross-validation technique for non-time-series problem. In this I learnt, that we decide the min ...
1
vote
1answer
29 views

Different hidden layer architectures deliver the same classification results, is that normal?

I have a data set with 600 data points with about 10 attributes (binary). The dataset has been normalized: Xnormalized = StandardScaler().fit_transform(X) The ...
0
votes
0answers
17 views

ValueError: Found input variables with inconsistent numbers of samples: [2, 921]

I want to apply K-Fold cross-validation to my neural network model, which looks like this: ...
2
votes
2answers
43 views

Hyperparameter optimization, ensembling instead of selecting with CV criteria

While burning CPUs performing a CV selection on a thin grid put on some hyperparameter space. I am using the `scikit-learn' API, for which the end result is a single point on the hyperparameter space, ...
1
vote
1answer
45 views

How to define the number of features to select in RFECV?

I am trying to work on feature selection stage for my dataset. I am a newbie to ML. I have around 60 columns and am trying to select top 15 features. I came to know about RFECV for which I wrote a ...
1
vote
1answer
131 views

New parameters in final training

I am training an Xgboost using 60% of my data and use 40% for testing. In the 60% of data, I use 5-fold validation to find the best number of trees. I find that the optimal number of trees is around ...
1
vote
0answers
13 views

How to do kfold cross-validation for multi-input models

The model is as below: ...
0
votes
1answer
30 views

K-Folds validation [duplicate]

Suppose we make a linear regression model on each of the 10 folds with the same number of features (say 2 for simplification) We will therefore have 10 sets of coefficients with the optimized values ​​...
2
votes
0answers
18 views

R train(method=“naive_bayes”) and naiveBayes() very different performance

I am an R novice and having some difficulty. I was hoping R would be a good (flexible, easy) way to do machine learning of textual data. A few years ago, I wrote a naive Bayesian classifier (from ...
2
votes
2answers
30 views

What is my training score the mean_train_score or mean_test_score?

I am using sklearn to train some models (random forest, decision tree). For the training I am using RandomsearchCV with Stratified k-fold as cross-validation. Then I make a predictions on the test set ...
1
vote
1answer
49 views

Multiple Linear Regression with k-fold Cross Validation

I would first like to create few multiple regression models based on if the models violate any multiple regression assumptions and how well it fits the training data. Then I would like to compare how ...
0
votes
1answer
14 views

Is it possible to change test and train data size when using crossvalind function with Kfold param?

I was looking at MATLAB Help and want to work with "crossvalind" function. It would two parameters that you can use it. If you use "HoldOut" you can define partition size of test and train data set ...
1
vote
2answers
42 views

Accuracy of KFold Cross Validation for Neural Network

I have a neural network that Im evaluating using 10 -Fold cross validation. The validation accuracy for a fold changes alot during training in the range of -+10% So for example the validation ...
1
vote
2answers
26 views

Cross validation vs leave one out

I have found the following definitions, but I don't really see the difference. cross validation Method for testing classification and prediction models. The data are randomly split into N partitions (...
1
vote
2answers
33 views

Acceptable variation in accuracy of each k fold when using K-Fold Cross Validation?

I have a relatively small dataset consisting of 1432 samples. I have trained a Random Forest Classifier and performed KFold CV. The results of running 10 Fold CV are as follows: ...
1
vote
0answers
18 views
1
vote
1answer
66 views

Cross-Validation: Repeated K-Fold/Group K-Fold

Repeated K-Fold vs Group K-Fold As per my understanding from sklearn docs Repeated K-Fold: RepeatedKFold repeats K-Fold n times. It can be used when one requires to run KFold n times, producing ...
1
vote
0answers
31 views

How to avoid bias when optimizing time-series model?

I have 4000 days of data. I am trying create a time-series model with parameters P to forecast the value of the target Y using the last N days of data. The parameters P include: lookback window for ...
0
votes
2answers
82 views

how to use standardization / standardscaler() for train and test?

At the moment I perform the following: ...
1
vote
0answers
20 views

Is kFoldLoss simply the average classification error?

I'm using a kNN classifier in MATLAB to classify ECG signals, with 5 fold cross validation I get almost 97% accuracy. However, I'm not entirely sure if this is truly the accuracy value. By default the ...
0
votes
1answer
19 views

which scoring function for validation_curve (regression)?

Is there any thumb of rule which scoring function should be used for e.g. the validation_curve? Atm I try to study the difference between several optimizers: ...
0
votes
0answers
10 views

Differnt loglikelihood values in RandomSearch/GridSearch and LDA . How to synchronize them?

I am doing a sensitivity analysis for some lda parameters in python. This is what I want to do: -Do RandomSearch to find optimal Parameters and therefore the optimal lda model. As Validation measure ...
0
votes
0answers
11 views

Standardizing in each fold - Learning Curve

Problem Description Hello, I have a classification problem and I want to perform cross validation (with hyper parameter tuning) in order to evaluate the generalization of my models. Basically the ...
0
votes
0answers
7 views

Hypothesis Test for Classifier, but without the original model

I just trained a number of DenseNet-based CNN models using an optimization technique that I created for my master dissertation. The problem is, my advisor wants me to perform a hypothesis test to ...
0
votes
2answers
79 views

K fold cross validation reduces accuracy

I am working on a machine learning classifier and when I arrive at the moment of dividing my data into training set and test set Iwant to confron two different approches. In one approch I just split ...
0
votes
0answers
17 views

CNN models comparison

I coded a 38 layer CNN and 8 layer CNN but there's something wrong in my 38 layer CNN, which doesn't learn anything. Not able to fugure out what's wrong. They were trained on CIFAR.
0
votes
1answer
42 views

Correctly evaluate model with oversampling and cross-validation

I'm dealing with a classic case of dataset with binary imbalanced target (event 3%, non event 97%). My idea is to apply some sort of sampling (over/under, SMOTE etc.) to address the issue. As I see, ...
0
votes
0answers
11 views

Track validation_curve during hyperparameter optimization

To study the influence of a single (hyper-)parameter, I use validation_curve: ...
0
votes
2answers
33 views

Is using cross validation on your entire dataset acceptable when dealing with a small sample size

Normally my practice includes using k-folds cross validation on a subset of my dataset and keep a final test set. When dealing with an exceptionally small dataset, is using cross validation on the ...
0
votes
2answers
62 views

cross validation issues

I have come here from this great answer. I have come across many approaches for using cross validation and the answer to the attached question is by far explaining it the best to me. My dilemma is ...
0
votes
3answers
40 views

Cross-fold validation done on whole dataset or training set?

I have a dataset of 77 samples with 302 features with two labels (0,1). I trained an SVM with gridsearch (cv=5) to perform binary classification. In one run of my script, I do a test-train split, ...
0
votes
1answer
34 views

Difference between validation and prediction

As a follow-up to Validate via predict() or via fit()? I wonder about the difference between validation and prediction. To keep it simple, I will refer to train, <...
1
vote
1answer
56 views

Hyperparameter tuning and cross validation

I have some confusion about proper usage of cross-validation to tune hyperparameters and evaluate estimator performance and generalizeability. As I understand it, this would be the process you ...
0
votes
0answers
83 views

Value Error: MSLE & CrossVal

I'm trying to run cross validation with mean squared log error with sklearn and getting the following error message: ...
0
votes
1answer
170 views

clarification on train, test and val and how to use/implement it

So far I think I understood the differences between the training, test and validation set. Basically it is like in this image: Training set: The data where the model is trained on Validation set: ...
0
votes
2answers
31 views

Drawing validation set from test set

I am building a 3 neural network models on dataset that is already separated to train and test sets. From my analysis, I found that this dataset has values on test set which don't exist in the train ...
1
vote
1answer
73 views

K-fold-cross-validation if training dataset is much smaller than test dataset?

I'm a beginner in machine learning and I have a special case in which I have only a small training dataset of about 500 images and a test dataset of 10,000 images. Does it still make sense to do a 10-...