Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

Filter by
Sorted by
Tagged with
2 votes
1 answer
17 views

Nested Cross-Validation with Small dataset

I am currently working with a small dataset (only 175 samples, 45 features) and have been reading on the proper way to cross-validate my model. I had started with a basic cross-validation using a grid ...
user avatar
0 votes
0 answers
9 views

How does exactly eval_set and RandomizedSearchCV work for LightGBM?

How does RandomizedSearchCV form the validation sets, while I also defined an evaluation set for LGBM? Is it formed from the train set I gave or how does the evaluation set comes into the validation? ...
user avatar
0 votes
0 answers
17 views

Lasso with cross validation - is training/test set split nedeed?

I am using LASSO regression to identify the most important variables for predicting an outcome. I have a relatively small sample (~1000 observations), and I was planning to use cross-validation to ...
user avatar
  • 1
0 votes
1 answer
32 views

Automated feature selection - Best practice to avoid data leakage?

This question relates generally to all automated feature selection approaches. In my particular scenario, we have a python package called tsfresh and multiclass classification. What has been done so ...
user avatar
  • 29
0 votes
1 answer
18 views

How to convert a string to callable

I have created a string array and called a method with that array along with Train, test data. The purpose of the method is to find Kfold results of each algorithm specifies in the array. Everything ...
user avatar
0 votes
0 answers
31 views

Leave One Group Out CV in Python

I'm trying to apply Leave One Group Out cross validation in python code by using sklearn's LeaveOneGroupOut() but I have a problem with defining the group split. So my data is consisted of 10 ...
user avatar
0 votes
0 answers
16 views

How to draw the ROC curve of a classifier evaluated with cross validation

I was wondering if anyone knows without with the R programming language, given a classifier evaluated by k fold cross validation, I can draw each of the ROC curves that are generated in each fold of ...
user avatar
  • 302
1 vote
0 answers
5 views

What to do if your adversarial validation show different distributions for an NLP problem?

I was trying to figure out if the test set from a competition is similar to the train set. This was done in a NLP competition, in which I had two columns, tweet and type, and I needed to predict the ...
user avatar
  • 161
0 votes
1 answer
23 views

Cross-validation for anomaly detection on time series data

I want to perform k-fold cross-validation for the setting where I have a training dataset consisting of a sequential time series that is fully benign and a test dataset (also a sequential time series)...
user avatar
3 votes
1 answer
96 views

How are parameters selected in cross-validation?

Suppose I'm training a linear regression model using k-fold cross-validation. I'm training K times each time with a different training and test data set. So each time I train, I get different ...
user avatar
  • 209
0 votes
1 answer
15 views

What does a leaf size of 1 in K-neighbors regression mean?

I am doing hyperparameter tuning + cross validation and I'm constantly getting that the optimal size of the leaf should be 1. Should I worry? Is this a sign of overfitting?
user avatar
  • 109
0 votes
0 answers
9 views

How does the datasampler widget's cross-validation option of Orange software work?

I use a datasampler widget to split a dataset (train and test) with the cross-validation selection. I wonder how it works because some points did not seem clear to me. Question 1: As seen in the ...
user avatar
0 votes
0 answers
19 views

stratified segment-grouped k-fold cross-validation

I have a music numerical data (2282 rows × 173 columns) to predict the target sad, happy, angry, relaxed. Now one of the attribute is segment_id and I want to group the data according to segment_id ...
user avatar
0 votes
0 answers
7 views

Difference between Jackknife vs bootsrap vs cross validation

I have doubts about the differences between these three methods and I would like to clarify the following: Main differences Advantages of one over the other Context of use of each method etc... If ...
user avatar
  • 302
0 votes
2 answers
19 views

do feature selection and model selection must share the same ratio between development set and test set?

As the title, after I performed a Feature Selection, is it mandatory to respect the same ratio (between development set and test set) in Model Selection?
user avatar
1 vote
2 answers
25 views

Cross Validation after using train-test to decide optimal algorithm to use?

I am interested in training different algorithms on a data set and observing performance metrics. Currently, my approach is to train different algorithms on train data, and then evaluate performance ...
user avatar
  • 11
0 votes
0 answers
11 views

my k-fold validation is giving a lot of 100% in the concatenated confusion matrix, is it because of overfitting?

The confusion matrix is a concatenated one from a 5-fold stratified cross-validation of my data set. I used rbf kernel for the svm classifier. Is it telling me the classifier is overfitting? Plus when ...
user avatar
  • 1
0 votes
0 answers
9 views

How to average different models produced when doing cross validation on a time series

I am currently cross validating a time series dataset and I have produced different models pertaining to the rolling window created. I read that I would have to average my models to get a generalized ...
user avatar
3 votes
1 answer
128 views

Interpreting ROC curves across k-fold cross-validation

I have used a MARS model (multivariate adaptive regression splines) and I have used k fold cross validation for the evaluation of the model, obtaining the following graph: How would be the ...
user avatar
  • 302
1 vote
0 answers
34 views

KFold splitting of time series

I have a fundamental question about train/test split for time series. Let me give a simple example to illustrate my question, which is actually related to a more complex problem. Example Suppose I ...
user avatar
0 votes
0 answers
11 views

How to understand CV

How to interpret cross_validation score in that case? While cross-val equeals minus
user avatar
0 votes
0 answers
13 views

How to use Splitting for startifying in sklearn for multiple files

I have csv data file for binary classification. I divided it into 5 multiple files and tried to apply the stratification technique so the class label has the same proportion for all the files. but I ...
user avatar
  • 175
1 vote
1 answer
41 views

Why is n-fold cross validation not expensive for K-nearest-neighbours?

I have on my uni lecture notes that one of the n-fold cross-validation disadvantages is that it is very expensive to train this because this could take a long time if the dataset is very large. But ...
user avatar
2 votes
1 answer
481 views

Why does gridsearchCV fit fail?

I already referred this post here but there is no answer. I am working on a binary classification using a random forest classifier. My dataset shape is (977,8) with 77:23 class proportion. My system ...
user avatar
  • 2,215
0 votes
1 answer
802 views

ValueError: The first argument to `Layer.call` must always be passed. for k Fold validation

Here is my model ...
user avatar
1 vote
0 answers
15 views

Regression tree cross validation confusing results

Setup I have implemented regression trees in go: full repository Using the full dataset and cost-complexity pruning, I get the following alphas and corresponding average residual (again, against the ...
user avatar
8 votes
2 answers
456 views

Why is the k-fold cross-validation needed?

I am using k-fold cross-validation but do not understand it's aim. Before splitting the data set in training and test data set, one usually randomizes the entries of the data set. Given the training ...
user avatar
0 votes
1 answer
53 views

What the difference between a flattening validation curve and one that increases again?

I know that we monitor the validation loss to investigate overfitting. I am familiar with the validation curve that first decreases and then increases again. The increasing part means the model starts ...
user avatar
  • 113
1 vote
1 answer
31 views

Brute-force feature selection and cross-validation

There is an existing score made of 10 parameters; each parameter is equally weighted & the total score is found by summing the score for each parameter. I want to try to reduce the number of ...
user avatar
  • 11
2 votes
3 answers
139 views

Should I use GridSearch CV for hyper-parameter tuning in a data-rich context?

My textbook states that k-fold cross-validation is a resampling technique that is useful for estimating generalization error in a data-poor setting. Ideally, if we had enough data, we would set aside ...
user avatar
0 votes
2 answers
93 views

binary classification pipeline to select threshold

There are quite a few questions regarding the optimisation of binary threshold in a classification problem. However, I haven't found a single end-to-end solution to this problem. In an existing ...
user avatar
  • 41
0 votes
1 answer
48 views

How to use a model after cross_validation in predicting a test data?

I want to do the following: train a model using cross-validation use the model for prediction (test dataset) check the algorithmic bias towards some features values I wonder if what I am doing is ...
user avatar
  • 13
1 vote
1 answer
18 views

Remedie for a stubborn recall result?

I was working on a project connected to predicting default on credit loan with 0-1 loss. The recall is a crucial measure that should be maximized in this case, while monitoring precision for sanity of ...
user avatar
0 votes
0 answers
9 views

How to train/test/validate hierachical classifiers?

I am writing an algorithm which allows to detect activities based on wearable data. I would like to try it out an hierachical approach (Local Classifier Per Parent Node structure). In the first level, ...
user avatar
  • 1
0 votes
0 answers
65 views

X_train, X_test, X_val, y_train, y_test, y_val for regression model with Keras or tensorflow?

How could I randomly split a data matrix and the corresponding label vector into a X_train, X_test, X_val, y_train, y_test, y_val for regression model with Keras or tensorflow..??? I have also added ...
user avatar
  • 1
0 votes
1 answer
30 views

Stacking neural nets with cross validation

I am trying to implement stacking model for a ML problem and having hard time figuring out the cross validation strategy. So far I have used 10-fold cross validation for all my models and would like ...
user avatar
0 votes
0 answers
18 views

How to correct a validation loss in a regression problem?

I've developed a graph neural network using PyTorch Geometric. My model looks like: ...
user avatar
0 votes
1 answer
26 views

Discrepancy between cross-validation and un-seen data predictions

I am facing an issue with an imbalanced dataset. The dataset contains 20% targets and 80% non-targets. I am expecting a confusion matrix below when I give un-seen data to the trained model. ...
user avatar
0 votes
0 answers
12 views

Feature engineering/selection within PySpark CrossValidator

Multiple sources I've read recently have argued that if using cross-validation to assist with model training/tuning, then aspects of the model development process such as feature selection should be ...
user avatar
  • 1
0 votes
1 answer
21 views

Relationship of Bias and size of dataset

I was reading the following book: http://www.feat.engineering/resampling.html where the author mentioned the below: Generally speaking, as the amount of data in the analysis set shrinks, the ...
user avatar
0 votes
0 answers
29 views

Does sklearn.pipeline have a single mechanism for cross-validation regardless of model API?

With a single standard interface (sklearn.pipeline) on top of different regressors, how do I use cross-validation? The example below uses two regressors with different internal cross-validation ...
user avatar
0 votes
3 answers
78 views

Need for cross-validation in KNN

I read that we need cross-validation in KNN algorithm as the K value that we have found from the TRAIN-TEST of KNN might not be generalizable on unseen data. The logic given was that, the TEST data ...
user avatar
0 votes
0 answers
31 views

Find train sample with features distribution similar to test

Assume that I have two datasets $Train$ and $Test$. And there is the problem illustrated below: there are different feature distribution between two datasets I want to find the train subset $A \...
user avatar
0 votes
1 answer
108 views

Custom vectorizer transformer in sklearn with cross validation

I created a custom transformer class called Vectorizer() that inherits from sklearn's ...
user avatar
  • 259
1 vote
1 answer
45 views

The behavior of the cross validation error and training error in underfitting case is not clear

I currently study the "Machine Learning" course on Coursera.org by Andrew Ng, it comes to a topic that discusses the performance of learning algorithms under different conditions. Here, we ...
user avatar
1 vote
1 answer
23 views

Random Forest Model Train, Save and Predict Later vs Train and Predict Right Away - Different Results

I tested two pieces of code and they delivered different results, which was quite unexpected. First piece of code is supposed to train models in a k-fold manner, preserve each one of these fitted ...
user avatar
  • 29
0 votes
0 answers
148 views

Split dataframe to a train and test sets with a cross validation of x%

I am working on a dataframe and need to split it into a training set and test set, with 90% for Cross-Validation training, and 10% for a final test set. The problem is that I do not know where to ...
user avatar
0 votes
0 answers
47 views

How to evaluate model accuracy at tail of empirical distribution?

I am making a nonlinear regression on stationary dependent variable and I want to precisely forecast extreme values of this variable. So when my model predicts extreme values I want them to be highly ...
user avatar
0 votes
2 answers
53 views

validation/test set uniqueness question

Hopefully a simple question, but it's a little unclear to me on how best to separate train/validate/test sets. I have say 100 examples of class A. I'm classifying text into either class A, which I ...
user avatar
  • 115
1 vote
0 answers
21 views

Interpreting XGBoost's results (when they span between [0,0.5])

I would like to classify sentences into one of two different categories. I trained a XGBoost model over a search grid with k-folds cross-validation. My data represents sentences, and features ...
user avatar

1
2 3 4 5
12