Questions tagged [cross-validation]
Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.
651
questions
1
vote
0
answers
21
views
Confused about use of random states for training models in scikit
I am new to ML and currently working on improving the accuracy of an MLPClassifier in scikit. My code looks like so
...
1
vote
1
answer
18
views
XGB find hyperparameters and then crossvalidation
I want to train an XGBoost model, and here's how I believe the process should go:
Step 1: Find the optimal hyperparameters using GridSearchCV.
Step 2: Evaluate the selected parameters.
My question is: ...
- 111
0
votes
1
answer
17
views
different result for k fold cross validation
For a small dataset that has less than 100 samples, I have run a model A. The result of R2 square for both test and train set is about 82%. But when I perform k fold cross validation on X, y, the ...
- 45
0
votes
1
answer
23
views
Nan result for one leave out cross validation
In this code, I get nan for the test result, what is the reason?
Average Train R^2: 0.504
Average Test R^2: nan
...
- 45
2
votes
1
answer
41
views
splitting small dataset into train and test set/
I have a very small dataset with only 70 samples an 18 features. The issue is that in this type of data set, splitting the data set into two parts, training and testing, causes a series of important ...
- 45
3
votes
1
answer
70
views
Why does scikit's cross-validation return a negative R^2 for my strongly correlated data
I have exactly the following preprocessed data in a small Pandas dataframe:
...
- 133
1
vote
2
answers
43
views
why k fold cross validation is bad only for knn?
The result of ten fold cross validation for all models that i run(like linear regression DCT, rf, etc) is very close to the result of test and train, but for knn regression it is very far from the ...
- 45
0
votes
0
answers
6
views
"Model" parameter in cross validation function in prophet
I have been using Prophet model to for demand forecasting. I have a general question about how I should be using the fitted model input to cross validation function.
...
0
votes
1
answer
22
views
Sklearn EstimatorCV vs GridSearchCV
sklearn has the following description for EstimatorCV estimators:
https://scikit-learn.org/stable/glossary.html#term-cross-validation-estimator
An estimator that has built-in cross-validation ...
0
votes
0
answers
26
views
How to choose thresholds to discretize target for binary classification
My group is using logistic regression to investigate the most predictive features in a dataset. Our target variable is actually a continuous variable that we discretized using two cutoff thresholds (...
0
votes
0
answers
18
views
How to Combine Cross-Validation Error and Ensemble Prediction Variance in Machine Learning?
I am working on a machine learning project where I use an ensemble model (Random Forest) and I want to accurately represent the prediction uncertainty. Specifically, I want to combine the cross-...
- 1
0
votes
1
answer
24
views
Averaging model performance across n-fold cross validation: MSE or R^2?
I'm comparing the performance of several models on the same data using cross-validation (holding out 1/n of the data as a test set, fitting the model on the remaining data, testing on the test set). I ...
2
votes
1
answer
34
views
Does it make sense that the performance of XG Boost varies dramatically from two machines holding all hyperparameters fixed?
I am hyperparameter tuning an xgboost model and I am finding that depending if I train the model locally on my machine vs on AWS sagemaker, I get quite different results.
Running cross-validation ...
- 121
0
votes
1
answer
89
views
Test Error is extremely higher than Training error after gridsearch and crossvalidation
I'm currently working on a machine learning project. It's a supervised learning problem. My goal is to predict for given data of an animal(keeping,size,weight,...) ingredients(energy,vitamine etc..). ...
2
votes
1
answer
21
views
Scoring function in cross-validation often left default
I'm a PhD student applying ML in microbiology. In research papers, the usual performance measure reported on classification models is ROC-AUC. But when I look at implementations, the scoring function ...
- 61
0
votes
1
answer
38
views
How do I identify overffiting when using GridSchearCV?
For context, I'm using Scikit Learn's GridSearchCV to find the best Hyperparameters of a Decision Tree.
I believe I understand Train, Validation, and Test sets and overfitting concepts when applied ...
0
votes
0
answers
19
views
How to use cross validation to select/evaluate model with probability score as the output?
Initially I was evaluating my models using cross_val with out-of-pocket metrics such as precision, recall, f1 score, etc, or with my own metrics defined in ...
- 21
0
votes
0
answers
15
views
Right Cross Validation Implementation (Regression)
I am very new to machine learning and i am starting to work my way up. I have made an implementation for cross validation which will be used with ensemble models later. I have made a pipeline in ...
- 1
1
vote
1
answer
14
views
Model evaluation approach allowing manual experimentation without data leakage
In supervised machine learning, are there any evaluation approaches beside using a fixed holdout test dataset, which allow me as a scientist to manually compare preprocessing approaches, without ...
- 111
0
votes
1
answer
22
views
Cross validation
I do not get why in For cross validation should I use training set, or whole dataset? the responses say that cross validation must be done exclusively on training set. Doesn't the methods (for example ...
1
vote
0
answers
7
views
Is GroupKFold needed if some samples have some of their feature values equal?
I am given a dataset $D$ of 10k enzyme-substrate complexes having a lock-key relationship, with each sample (complex) being characterized by enzyme features $x_e$ and substrate features $x_s$. That is,...
- 191
0
votes
0
answers
18
views
How does hyperparameter tuning work for constructing/choosing a final model using Nested Cross validation?
I want to determine if XGBoost is better than random forest or logistic regression for building a binary classification model. The model will be a composite model, with a feature selection model to ...
0
votes
0
answers
28
views
If I do cross validation do I need to refit the model?
I am making a dual process. I have an initial dataset in which I train (fit) a model, then I do cross validation to get results. Until now everything normal, but additional to that, I create a new ...
0
votes
1
answer
12
views
Doing correctly paired-trial validation
In paired-trial validation, a statistical (ML) models are trained on $n$ datasets separately and then applied to other datasets, as a way of estimating the generalization of the models obtained. ...
- 103
0
votes
0
answers
30
views
I have a poor understanding of nested cv and generalization
I'm not sure if I understand the purpose and generalization of 'nested cv' correctly.
I found information online that the purpose of nestd cv is to be able to correctly estimate generalization error. ...
- 13
0
votes
0
answers
11
views
How to assess the stability of a DL model, after using k-fold cross-validation for hyperparameter tuning
I've recently completed the training of a deep learning model for a classification task, using a process that involves k-fold cross-validation for hyperparameter tuning
Initially, I have divided my ...
- 1
0
votes
1
answer
54
views
As a result of cross-validation, the difference between the ideal auc values of the train set and the test set
In the attached figure, the x-axis is the number features of s removed, and the y-axis is the average auc score over 10 CVs.
I want to choose the point with the highest score while avoiding ...
- 13
0
votes
0
answers
11
views
How are the cross validation and training processes interlinked here?
Please consider the code given below.
...
- 101
0
votes
0
answers
25
views
5-fold cross validation in R: getting error Age variable different lengths
I am tasked to do a 5-fold cross validation for my R grad course with pga golf data. I continually get an error for a certain variable, Age, saying different lengths. Here is the error code:
...
0
votes
0
answers
16
views
Does it make sense to do hp tuning for a Random Forest for top k precision or recall?
I've trained an RF with a binary classification task that achieves mediocre performance. However, they way it is intended to be used would have end-users look only at predictions with high scores (...
0
votes
0
answers
18
views
How do you actually train a model to make predictions on new inputed data?
I created a system for my thesis, this system can predict whether something will succeed or fail. I use k-fold cross validation to see the performance, then the model that will be used to predict new ...
0
votes
0
answers
13
views
Are hyperparameters trained when k-fold cross validation is applied
I just started working with k fold CV and am a little bit confused about the topic mentioned above. How are my hyperparameters tuned if I use a k fold approach to train my model. What I read so far is ...
0
votes
1
answer
39
views
Selecting optimal regression model using cross validation
I have a logistic mixed model (lme4 package in R). I want to assess whether participants scores on the measures 'sumspq', 'sumpdi', and 'sumcaps' significantly affect the difference in performance ...
- 103
0
votes
0
answers
45
views
Pycaret cross validation scores are way lower than unseen test set scores
How come the cross-validation model scores are much lower than the model scores on an unseen dataset?
...
- 1
1
vote
2
answers
177
views
Why does undersampling before cross-validation lead to leakage?
I came across the paper "Leakage and the Reproducibility Crisis in ML-based Science" by Sayash Kapoor and Arvind Narayanan, wherein the authors argue that both over- and under-sampling the ...
- 13
0
votes
3
answers
62
views
Why the one validation score is lower than the other sections of cross validation
I was working on RandomForestClassifier and doing hyperparameter tuning. But something caught my attention. I always get a lower validation value in the 2nd part of Cross Validation. Here is the code:
...
0
votes
0
answers
15
views
Stable test in online time series forecasting problem
I have a Time Series Forecasting problem. You can think of it as predicting the daily closing prices of Apple stocks. My data is divided into 4-day segments, and the forecasting is based on predicting ...
- 63
0
votes
1
answer
274
views
Is Repeated K-Fold Cross Validation Enough to Evaluate a Machine Learning Model?
I am training models with a small dataset (around 800 observations) and I am using Repeated K-Fold cross validation to evaluate the models.
Initially, i am using the same cross validation for ...
0
votes
0
answers
71
views
XGBoost Classifier Evaluation Confusion on New Dataset Despite High Cross-Validation Scores
I have built an XGBoost classifier model with 90 features, trained on a dataset containing 760k samples. I took great care to separate the labels from the features in both the training and testing ...
- 1
0
votes
1
answer
26
views
How to pass a Dataframe as train dataframe and another dataframe as Validation to GridSearchCV
I'm a programmer who tries to find he's way into ML world. so the Question might be basic.
i have data from years 2010-2019. Now i'm trying to test different parameters on gradient boosting regression ...
- 101
1
vote
1
answer
132
views
Trying to understand Nested k-fold CV in a paper
I fully understand the data partition in a nested k-fold CV. But reading this:
Within each outer fold, the best performing model was selected based on mean root mean squared error (RMSE) over the
...
0
votes
2
answers
40
views
Optimal Data Split
I have a multiclass problem (3 classes) that looks to predict if someone will buy a product, neutral or not. I have initial features of in-app activity data such as likes, share, bookmark, share, ...
0
votes
1
answer
39
views
Cross validation and train_test_split
I am building a class that follows the workflow:
Model Selection and Fitting
The class accepts a list of models and their respective hyperparameter grids. It then performs a standard fitting process ...
0
votes
3
answers
538
views
For cross validation should I use training set, or whole dataset?
I'm new to data science and I have a problem understanding what dataset to use when using cross validation for model evaluation.
Let's say I have two models: LogisticRegression and ...
0
votes
1
answer
25
views
Challenges in Predicting Molecule Activity
I want to share a concern I have. I want to obtain a machine learning model that can predict whether a molecule exhibits biological activity. For this purpose, I have a set of molecules that do ...
0
votes
0
answers
58
views
ML model to predict CPU utilization of a server given x amount of tasks
I have comprehensive data points of what the CPU utilization of a server is when x amount of jobs are running, let's say the server is using 40% CPU util time=x and there are 4 jobs running.
The ...
0
votes
0
answers
57
views
Build a model with cross-validation on entire dataset to learn insights?
Goal : Use XGBoost regression to learn insights from data. Prediction or forecasting not needed.
Hypothesis : If the model fits the entire dataset well, it can maybe capture its "physics" in ...
1
vote
1
answer
98
views
integration of Feature Selection in Pipeline
I have noticed integrating feature selection in a pipeline alters results.
Pipeline 1 gives slightly different results with pipeline 2. Why should this be so?
Pipeline 2
...
- 113
0
votes
1
answer
437
views
Can I use GridSearchCV.best_score_ for evaluation of model performance?
Scikit-learn page on Grid Search says:
Model selection by evaluating various parameter settings can be seen
as a way to use the labeled data to “train” the parameters of the
grid.
When evaluating the ...
- 103
1
vote
0
answers
210
views
What is the best way to combine cross-validation and bootstrapping for one application?
We intend to model data with non-parametric covariate splines and we would like to understand the uncertainty of the parameter estimates/response estimates.
Currently, we use cross-validation to model ...
- 111