Skip to main content

Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

Filter by
Sorted by
Tagged with
0 votes
0 answers
14 views

How to Combine Cross-Validation Error and Ensemble Prediction Variance in Machine Learning?

I am working on a machine learning project where I use an ensemble model (Random Forest) and I want to accurately represent the prediction uncertainty. Specifically, I want to combine the cross-...
x H's user avatar
  • 1
0 votes
1 answer
11 views

Averaging model performance across n-fold cross validation: MSE or R^2?

I'm comparing the performance of several models on the same data using cross-validation (holding out 1/n of the data as a test set, fitting the model on the remaining data, testing on the test set). I ...
Leo Selker's user avatar
2 votes
1 answer
27 views

Does it make sense that the performance of XG Boost varies dramatically from two machines holding all hyperparameters fixed?

I am hyperparameter tuning an xgboost model and I am finding that depending if I train the model locally on my machine vs on AWS sagemaker, I get quite different results. Running cross-validation ...
Luca Guarro's user avatar
0 votes
1 answer
78 views

Test Error is extremely higher than Training error after gridsearch and crossvalidation

I'm currently working on a machine learning project. It's a supervised learning problem. My goal is to predict for given data of an animal(keeping,size,weight,...) ingredients(energy,vitamine etc..). ...
Marco Cotrotzo's user avatar
1 vote
1 answer
18 views

Scoring function in cross-validation often left default

I'm a PhD student applying ML in microbiology. In research papers, the usual performance measure reported on classification models is ROC-AUC. But when I look at implementations, the scoring function ...
alepfu's user avatar
  • 51
0 votes
1 answer
25 views

How do I identify overffiting when using GridSchearCV?

For context, I'm using Scikit Learn's GridSearchCV to find the best Hyperparameters of a Decision Tree. I believe I understand Train, Validation, and Test sets and overfitting concepts when applied ...
Lisana Daniel's user avatar
0 votes
0 answers
17 views

How to use cross validation to select/evaluate model with probability score as the output?

Initially I was evaluating my models using cross_val with out-of-pocket metrics such as precision, recall, f1 score, etc, or with my own metrics defined in ...
szheng's user avatar
  • 21
0 votes
0 answers
13 views

Right Cross Validation Implementation (Regression)

I am very new to machine learning and i am starting to work my way up. I have made an implementation for cross validation which will be used with ensemble models later. I have made a pipeline in ...
Guhan's user avatar
  • 1
1 vote
1 answer
14 views

Model evaluation approach allowing manual experimentation without data leakage

In supervised machine learning, are there any evaluation approaches beside using a fixed holdout test dataset, which allow me as a scientist to manually compare preprocessing approaches, without ...
thomas8wp's user avatar
  • 111
0 votes
1 answer
20 views

Cross validation

I do not get why in For cross validation should I use training set, or whole dataset? the responses say that cross validation must be done exclusively on training set. Doesn't the methods (for example ...
Curious student's user avatar
1 vote
0 answers
7 views

Is GroupKFold needed if some samples have some of their feature values equal?

I am given a dataset $D$ of 10k enzyme-substrate complexes having a lock-key relationship, with each sample (complex) being characterized by enzyme features $x_e$ and substrate features $x_s$. That is,...
ado sar's user avatar
  • 191
0 votes
0 answers
11 views

How does hyperparameter tuning work for constructing/choosing a final model using Nested Cross validation?

I want to determine if XGBoost is better than random forest or logistic regression for building a binary classification model. The model will be a composite model, with a feature selection model to ...
reuben george's user avatar
0 votes
0 answers
22 views

If I do cross validation do I need to refit the model?

I am making a dual process. I have an initial dataset in which I train (fit) a model, then I do cross validation to get results. Until now everything normal, but additional to that, I create a new ...
Curious student's user avatar
0 votes
1 answer
12 views

Doing correctly paired-trial validation

In paired-trial validation, a statistical (ML) models are trained on $n$ datasets separately and then applied to other datasets, as a way of estimating the generalization of the models obtained. ...
Roger V.'s user avatar
  • 103
0 votes
0 answers
28 views

I have a poor understanding of nested cv and generalization

I'm not sure if I understand the purpose and generalization of 'nested cv' correctly. I found information online that the purpose of nestd cv is to be able to correctly estimate generalization error. ...
JAE's user avatar
  • 13
0 votes
0 answers
10 views

How to assess the stability of a DL model, after using k-fold cross-validation for hyperparameter tuning

I've recently completed the training of a deep learning model for a classification task, using a process that involves k-fold cross-validation for hyperparameter tuning Initially, I have divided my ...
o'hara's user avatar
  • 1
0 votes
1 answer
34 views

As a result of cross-validation, the difference between the ideal auc values ​of the train set and the test set

In the attached figure, the x-axis is the number features of s removed, and the y-axis is the average auc score over 10 CVs. I want to choose the point with the highest score while avoiding ...
JAE's user avatar
  • 13
0 votes
0 answers
11 views

How are the cross validation and training processes interlinked here?

Please consider the code given below. ...
Masroor's user avatar
  • 101
0 votes
0 answers
18 views

5-fold cross validation in R: getting error Age variable different lengths

I am tasked to do a 5-fold cross validation for my R grad course with pga golf data. I continually get an error for a certain variable, Age, saying different lengths. Here is the error code: ...
Heather S.'s user avatar
0 votes
0 answers
12 views

Does it make sense to do hp tuning for a Random Forest for top k precision or recall?

I've trained an RF with a binary classification task that achieves mediocre performance. However, they way it is intended to be used would have end-users look only at predictions with high scores (...
ds_banter's user avatar
0 votes
0 answers
17 views

How do you actually train a model to make predictions on new inputed data?

I created a system for my thesis, this system can predict whether something will succeed or fail. I use k-fold cross validation to see the performance, then the model that will be used to predict new ...
Agus Krisna Perdana's user avatar
0 votes
0 answers
11 views

Are hyperparameters trained when k-fold cross validation is applied

I just started working with k fold CV and am a little bit confused about the topic mentioned above. How are my hyperparameters tuned if I use a k fold approach to train my model. What I read so far is ...
Sisoviromol's user avatar
0 votes
1 answer
38 views

Selecting optimal regression model using cross validation

I have a logistic mixed model (lme4 package in R). I want to assess whether participants scores on the measures 'sumspq', 'sumpdi', and 'sumcaps' significantly affect the difference in performance ...
SilvaC's user avatar
  • 103
0 votes
0 answers
26 views

Pycaret cross validation scores are way lower than unseen test set scores

How come the cross-validation model scores are much lower than the model scores on an unseen dataset? ...
yoavf's user avatar
  • 1
1 vote
2 answers
137 views

Why does undersampling before cross-validation lead to leakage?

I came across the paper "Leakage and the Reproducibility Crisis in ML-based Science" by Sayash Kapoor and Arvind Narayanan, wherein the authors argue that both over- and under-sampling the ...
Viades's user avatar
  • 13
0 votes
3 answers
62 views

Why the one validation score is lower than the other sections of cross validation

I was working on RandomForestClassifier and doing hyperparameter tuning. But something caught my attention. I always get a lower validation value in the 2nd part of Cross Validation. Here is the code: ...
Emir Kutsal's user avatar
0 votes
0 answers
13 views

Stable test in online time series forecasting problem

I have a Time Series Forecasting problem. You can think of it as predicting the daily closing prices of Apple stocks. My data is divided into 4-day segments, and the forecasting is based on predicting ...
Angerato's user avatar
0 votes
1 answer
198 views

Is Repeated K-Fold Cross Validation Enough to Evaluate a Machine Learning Model?

I am training models with a small dataset (around 800 observations) and I am using Repeated K-Fold cross validation to evaluate the models. Initially, i am using the same cross validation for ...
codenoob1211's user avatar
0 votes
0 answers
53 views

XGBoost Classifier Evaluation Confusion on New Dataset Despite High Cross-Validation Scores

I have built an XGBoost classifier model with 90 features, trained on a dataset containing 760k samples. I took great care to separate the labels from the features in both the training and testing ...
oklen's user avatar
  • 1
0 votes
1 answer
24 views

How to pass a Dataframe as train dataframe and another dataframe as Validation to GridSearchCV

I'm a programmer who tries to find he's way into ML world. so the Question might be basic. i have data from years 2010-2019. Now i'm trying to test different parameters on gradient boosting regression ...
Mostafa Bouzari's user avatar
1 vote
1 answer
112 views

Trying to understand Nested k-fold CV in a paper

I fully understand the data partition in a nested k-fold CV. But reading this: Within each outer fold, the best performing model was selected based on mean root mean squared error (RMSE) over the ...
Amirhossein Rezaei's user avatar
0 votes
0 answers
85 views

Classification Threshold Optimization after GridSearchCV

In my machine learning problem I am using a CNN to classify images. Since my dataset is imbalanced I want to perform classification probability threshold tuning so I can find the optimal balance ...
Throwaway123's user avatar
0 votes
2 answers
39 views

Optimal Data Split

I have a multiclass problem (3 classes) that looks to predict if someone will buy a product, neutral or not. I have initial features of in-app activity data such as likes, share, bookmark, share, ...
easymoneysniper's user avatar
0 votes
0 answers
14 views

Test accuracy plateaus when increasing max_depth -> inf

I've built a Random Forest model that classifies into four categories based on around 10 input features. To test the accuracy, I performed 5-fold stratified cross validation using the ...
okjdlsksjdwi's user avatar
0 votes
1 answer
34 views

Cross validation and train_test_split

I am building a class that follows the workflow: Model Selection and Fitting The class accepts a list of models and their respective hyperparameter grids. It then performs a standard fitting process ...
Guilherme Raibolt's user avatar
0 votes
3 answers
367 views

For cross validation should I use training set, or whole dataset?

I'm new to data science and I have a problem understanding what dataset to use when using cross validation for model evaluation. Let's say I have two models: LogisticRegression and ...
Michał Jurzak's user avatar
0 votes
1 answer
24 views

Challenges in Predicting Molecule Activity

I want to share a concern I have. I want to obtain a machine learning model that can predict whether a molecule exhibits biological activity. For this purpose, I have a set of molecules that do ...
Yasser Hayek's user avatar
0 votes
0 answers
57 views

ML model to predict CPU utilization of a server given x amount of tasks

I have comprehensive data points of what the CPU utilization of a server is when x amount of jobs are running, let's say the server is using 40% CPU util time=x and there are 4 jobs running. The ...
cpuUtilServerHelp's user avatar
0 votes
0 answers
55 views

Build a model with cross-validation on entire dataset to learn insights?

Goal : Use XGBoost regression to learn insights from data. Prediction or forecasting not needed. Hypothesis : If the model fits the entire dataset well, it can maybe capture its "physics" in ...
cwanderroycbooks's user avatar
0 votes
0 answers
13 views

Can we apply cross validation bias correction to explanatory variable selection as well?

As shown in the paper below, several methods have been suggested for bias correction for cross validation. For example, Tibshirani-Tibshirani method and BBC-CV method. These are known to significantly ...
x H's user avatar
  • 1
1 vote
1 answer
82 views

integration of Feature Selection in Pipeline

I have noticed integrating feature selection in a pipeline alters results. Pipeline 1 gives slightly different results with pipeline 2. Why should this be so? Pipeline 2 ...
wwnde's user avatar
  • 113
0 votes
1 answer
282 views

Can I use GridSearchCV.best_score_ for evaluation of model performance?

Scikit-learn page on Grid Search says: Model selection by evaluating various parameter settings can be seen as a way to use the labeled data to “train” the parameters of the grid. When evaluating the ...
Charlie's user avatar
  • 103
1 vote
0 answers
189 views

What is the best way to combine cross-validation and bootstrapping for one application?

We intend to model data with non-parametric covariate splines and we would like to understand the uncertainty of the parameter estimates/response estimates. Currently, we use cross-validation to model ...
Stan Tendijck's user avatar
0 votes
0 answers
19 views

Validation error less than training error

entire project link: Github Repository In a classification task using Neural Network, I computed the fraction of misclassification as an error. And I am getting a validation error less than the ...
Dipen Pandit's user avatar
0 votes
1 answer
92 views

Why does my custom rmsle_loss produce negative scores during cross-validation?

I have a pipeline and scorer that produces some unexpected behaviors during cross-validation. During cross validation the pipeline produces all negative scores, but from the definition from rmsle_loss ...
Tim's user avatar
  • 5
0 votes
0 answers
38 views

Surprise NMF object is not callable

I am building a recommender system using the Sushi Preference Dataset and the NMF (Non-negative Matrix Factorization) model. I am implementing the same using the Surprise library. I want to use ...
Sumant Chopde's user avatar
1 vote
2 answers
46 views

which hyperparameters are returned as best in cross validation?

The description on the RandomizedSearchCV says this about best hyperparameters : "Estimator that was chosen by the search, i.e. estimator which gave highest ...
pppp_prs's user avatar
0 votes
1 answer
48 views

error when using KFold() and roc_auc metric

why cross_val_score(pipe,X,y,scoring="roc_auc",cv=StratifiedKFold()) works just fine and when using KFold() like ...
jxqbbb's user avatar
  • 51
0 votes
0 answers
26 views

Unsupervised clustering approach validated using internal data

I have used mclust package in R software for unsupervised clustering approach and choosing the clustering result according to the minimum BIC value. Can I use the cross validation method and calculate ...
許乃偉's user avatar
0 votes
1 answer
50 views

Is this the best method for comparing different approaches nd selecting the best model in machine learning?

My objective is to experiment with various approaches for different algorithms, identify the best approach for each algorithm, and subsequently determine the best overall algorithm from among these ...
Salah Amani's user avatar

1
2 3 4 5
13