Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

5-fold cross validation in R: getting error Age variable different lengths

I am tasked to do a 5-fold cross validation for my R grad course with pga golf data. I continually get an error for a certain variable, Age, saying different lengths. Here is the error code: ...
Heather S.'s user avatar
0 votes
0 answers
24 views

Cross validation for a high-noise problem

I am training a binary classification model for a high noise task which I want to optimise using $k$-fold cross-validation. My main issue is that the test error fluctuates a lot with the train/test ...
Julian's user avatar
  • 101
0 votes
0 answers
22 views

Which data should I test during the cross validation process?

I am developing a classification application by fine tuning the pre-train bert model. First I divide my data by 2. Train and test. Then, I train my train data in 10 repeats and 10 folds with the cross ...
Nemo's user avatar
  • 1
0 votes
0 answers
11 views

Does it make sense to do hp tuning for a Random Forest for top k precision or recall?

I've trained an RF with a binary classification task that achieves mediocre performance. However, they way it is intended to be used would have end-users look only at predictions with high scores (...
ds_banter's user avatar
0 votes
0 answers
15 views

How do you actually train a model to make predictions on new inputed data?

I created a system for my thesis, this system can predict whether something will succeed or fail. I use k-fold cross validation to see the performance, then the model that will be used to predict new ...
Agus Krisna Perdana's user avatar
0 votes
0 answers
10 views

Are hyperparameters trained when k-fold cross validation is applied

I just started working with k fold CV and am a little bit confused about the topic mentioned above. How are my hyperparameters tuned if I use a k fold approach to train my model. What I read so far is ...
Sisoviromol's user avatar
0 votes
1 answer
30 views

Selecting optimal regression model using cross validation

I have a logistic mixed model (lme4 package in R). I want to assess whether participants scores on the measures 'sumspq', 'sumpdi', and 'sumcaps' significantly affect the difference in performance ...
SilvaC's user avatar
  • 103
0 votes
0 answers
19 views

Pycaret cross validation scores are way lower than unseen test set scores

How come the cross-validation model scores are much lower than the model scores on an unseen dataset? ...
yoavf's user avatar
  • 1
1 vote
2 answers
66 views

Why does undersampling before cross-validation lead to leakage?

I came across the paper "Leakage and the Reproducibility Crisis in ML-based Science" by Sayash Kapoor and Arvind Narayanan, wherein the authors argue that both over- and under-sampling the ...
Viades's user avatar
  • 13
0 votes
3 answers
61 views

Why the one validation score is lower than the other sections of cross validation

I was working on RandomForestClassifier and doing hyperparameter tuning. But something caught my attention. I always get a lower validation value in the 2nd part of Cross Validation. Here is the code: ...
Emir Kutsal's user avatar
0 votes
0 answers
12 views

Stable test in online time series forecasting problem

I have a Time Series Forecasting problem. You can think of it as predicting the daily closing prices of Apple stocks. My data is divided into 4-day segments, and the forecasting is based on predicting ...
Angerato's user avatar
0 votes
1 answer
106 views

Is Repeated K-Fold Cross Validation Enough to Evaluate a Machine Learning Model?

I am training models with a small dataset (around 800 observations) and I am using Repeated K-Fold cross validation to evaluate the models. Initially, i am using the same cross validation for ...
codenoob1211's user avatar
0 votes
0 answers
33 views

XGBoost Classifier Evaluation Confusion on New Dataset Despite High Cross-Validation Scores

I have built an XGBoost classifier model with 90 features, trained on a dataset containing 760k samples. I took great care to separate the labels from the features in both the training and testing ...
oklen's user avatar
  • 1
0 votes
1 answer
22 views

How to pass a Dataframe as train dataframe and another dataframe as Validation to GridSearchCV

I'm a programmer who tries to find he's way into ML world. so the Question might be basic. i have data from years 2010-2019. Now i'm trying to test different parameters on gradient boosting regression ...
Mostafa Bouzari's user avatar
1 vote
1 answer
94 views

Trying to understand Nested k-fold CV in a paper

I fully understand the data partition in a nested k-fold CV. But reading this: Within each outer fold, the best performing model was selected based on mean root mean squared error (RMSE) over the ...
Amirhossein Rezaei's user avatar
0 votes
0 answers
58 views

Classification Threshold Optimization after GridSearchCV

In my machine learning problem I am using a CNN to classify images. Since my dataset is imbalanced I want to perform classification probability threshold tuning so I can find the optimal balance ...
Throwaway123's user avatar
0 votes
2 answers
35 views

Optimal Data Split

I have a multiclass problem (3 classes) that looks to predict if someone will buy a product, neutral or not. I have initial features of in-app activity data such as likes, share, bookmark, share, ...
Marc Atanante's user avatar
0 votes
0 answers
12 views

Test accuracy plateaus when increasing max_depth -> inf

I've built a Random Forest model that classifies into four categories based on around 10 input features. To test the accuracy, I performed 5-fold stratified cross validation using the ...
okjdlsksjdwi's user avatar
0 votes
1 answer
31 views

Cross validation and train_test_split

I am building a class that follows the workflow: Model Selection and Fitting The class accepts a list of models and their respective hyperparameter grids. It then performs a standard fitting process ...
Guilherme Raibolt's user avatar
0 votes
3 answers
193 views

For cross validation should I use training set, or whole dataset?

I'm new to data science and I have a problem understanding what dataset to use when using cross validation for model evaluation. Let's say I have two models: LogisticRegression and ...
Michał Jurzak's user avatar
0 votes
1 answer
24 views

Challenges in Predicting Molecule Activity

I want to share a concern I have. I want to obtain a machine learning model that can predict whether a molecule exhibits biological activity. For this purpose, I have a set of molecules that do ...
Yasser Hayek's user avatar
0 votes
0 answers
43 views

ML model to predict CPU utilization of a server given x amount of tasks

I have comprehensive data points of what the CPU utilization of a server is when x amount of jobs are running, let's say the server is using 40% CPU util time=x and there are 4 jobs running. The ...
cpuUtilServerHelp's user avatar
0 votes
0 answers
53 views

Build a model with cross-validation on entire dataset to learn insights?

Goal : Use XGBoost regression to learn insights from data. Prediction or forecasting not needed. Hypothesis : If the model fits the entire dataset well, it can maybe capture its "physics" in ...
cwanderroycbooks's user avatar
0 votes
0 answers
12 views

Can we apply cross validation bias correction to explanatory variable selection as well?

As shown in the paper below, several methods have been suggested for bias correction for cross validation. For example, Tibshirani-Tibshirani method and BBC-CV method. These are known to significantly ...
x H's user avatar
  • 1
1 vote
1 answer
57 views

integration of Feature Selection in Pipeline

I have noticed integrating feature selection in a pipeline alters results. Pipeline 1 gives slightly different results with pipeline 2. Why should this be so? Pipeline 2 ...
wwnde's user avatar
  • 113
0 votes
1 answer
115 views

Can I use GridSearchCV.best_score_ for evaluation of model performance?

Scikit-learn page on Grid Search says: Model selection by evaluating various parameter settings can be seen as a way to use the labeled data to “train” the parameters of the grid. When evaluating the ...
Charlie's user avatar
  • 103
1 vote
0 answers
148 views

What is the best way to combine cross-validation and bootstrapping for one application?

We intend to model data with non-parametric covariate splines and we would like to understand the uncertainty of the parameter estimates/response estimates. Currently, we use cross-validation to model ...
Stan Tendijck's user avatar
0 votes
0 answers
10 views

Validation error less than training error

entire project link: Github Repository In a classification task using Neural Network, I computed the fraction of misclassification as an error. And I am getting a validation error less than the ...
Dipen Pandit's user avatar
0 votes
1 answer
55 views

Why does my custom rmsle_loss produce negative scores during cross-validation?

I have a pipeline and scorer that produces some unexpected behaviors during cross-validation. During cross validation the pipeline produces all negative scores, but from the definition from rmsle_loss ...
Tim's user avatar
  • 5
0 votes
0 answers
34 views

Surprise NMF object is not callable

I am building a recommender system using the Sushi Preference Dataset and the NMF (Non-negative Matrix Factorization) model. I am implementing the same using the Surprise library. I want to use ...
Sumant Chopde's user avatar
1 vote
2 answers
41 views

which hyperparameters are returned as best in cross validation?

The description on the RandomizedSearchCV says this about best hyperparameters : "Estimator that was chosen by the search, i.e. estimator which gave highest ...
pppp_prs's user avatar
0 votes
1 answer
26 views

error when using KFold() and roc_auc metric

why cross_val_score(pipe,X,y,scoring="roc_auc",cv=StratifiedKFold()) works just fine and when using KFold() like ...
jxqbbb's user avatar
  • 51
0 votes
0 answers
26 views

Unsupervised clustering approach validated using internal data

I have used mclust package in R software for unsupervised clustering approach and choosing the clustering result according to the minimum BIC value. Can I use the cross validation method and calculate ...
許乃偉's user avatar
0 votes
0 answers
22 views

Is subsequent cross validation on the same dataset biased?

I am training ML regression models to predict financial returns in a high frequency trading environment. I have 1 time-series of intraday data for 40 years for 1 individual security at the moment. I ...
Cap_F's user avatar
  • 1
0 votes
1 answer
44 views

Is this the best method for comparing different approaches nd selecting the best model in machine learning?

My objective is to experiment with various approaches for different algorithms, identify the best approach for each algorithm, and subsequently determine the best overall algorithm from among these ...
Salah Amani's user avatar
1 vote
1 answer
121 views

scikit-learn cross_val_score randomness

Does cross_val_score in scikit-learn split the data consistently or randomly? I noticed that cross_val_score lacks a random_state parameter, but the documentation mentions stratified k-fold cross-...
jxqbbb's user avatar
  • 51
0 votes
0 answers
13 views

Fairness post-processing changing threshold and margin

I have a doubt, I want to use fairness methods and I see the tutorial they split test into test and validation and use validation to determine the best threshold and margin for predictions. But I ...
Esmeralda Ruiz Pujadas's user avatar
0 votes
0 answers
85 views

Visualize Catboost and XGBoost training process + Cross Validation

I want to optimize Catboost and XGBoost models and visualize this process such that: Use 3-fold cross-validation Use my own pre-processing pipeline (Missing value imputation, over- or undersampling) ...
Ars ML's user avatar
  • 61
0 votes
3 answers
198 views

How to properly do a k-fold cross validation?

I am trying to solve binary classification problem using deep neural networks. I want to compare different approaches (model architectures) and I have no hyperparameters which I want to tune. So my ...
dmasny's user avatar
  • 13
0 votes
0 answers
74 views

Walk forward cross-validation with Optuna and deepar in pytorch forecasting

I want to perform 3 splits walk forward cross validation with expanding training set for the deepar model from the pytorch forecasting framework. When I do walk forward validation, I also want to do ...
Jose_Peeterson's user avatar
0 votes
1 answer
28 views

Flow of machine learning model including code

I'm towards the completion of my first data science project that will go into my GitHub portfolio. I'll be happy for some clarification regarding the machine learning models section: I got a little ...
Sigal Cohen's user avatar
1 vote
1 answer
62 views

What is the point of final test set in K-fold cross-validation?

I am carrying out logistic regression for my binary classification problem, and I have validated the model with kfold cross-validation (k=10). I don't understand why I need to have a final test set, ...
Karoline Teller's user avatar
0 votes
0 answers
45 views

Should I use mean, standard deviation, or coefficient of variation in cross-validation?

I'm using cross-validation and calculating the AUCs, and then calculating the mean, the standard deviation(SD), and taking the same standard deviation and dividing it by the same mean to calculate the ...
Maxi's user avatar
  • 89
0 votes
0 answers
32 views

Is this a valid cross validation approach to choose hyperparameters and get a good estimate of model performace?

I am using lightGBM on time series data. I first split my data set into 10% folds. The last fold is used as a test set. For each choice of hyperparameters I first use 6 folds to train, then predict on ...
user3376601's user avatar
0 votes
0 answers
21 views

Can I firstly use LazyPredict and RandomizedSearchCV later?

If I have a training a test set, X_train and X_test, can I firstly use LazyPredict to find the best model and later tuning that model's parameters using RandomizedSearchCV? My doubt is that the ...
Flavio Brienza's user avatar
1 vote
2 answers
330 views

RandomizedSearchcv(n_iter=10) doesnt stop after training 10 models

I am using RandomizedSearchcv for hyperparameter optimization. When I run the model, it shows the scores for each model training. The problem is, it trains way more than 10 models when in fact I ...
Mehmet Deniz's user avatar
0 votes
1 answer
71 views

How do I know If my regression model is underfitting?

How do we evaluate the performance of a regression model with a certain RMSE given that a domain knowledge performance metric is not present? Maybe MAPE is one way of comparing the performance of my ...
Mehmet Deniz's user avatar
0 votes
0 answers
35 views

Common cross-validation code: why does it work?

The following Python code is common practice when creating a folds column for multi-label stratified k-fold cross-validation: ...
tossimmar's user avatar
  • 101
0 votes
1 answer
78 views

Fit multiple models e.g classifiers -> stacking -> calibration without data-leak or getting too many datasets

I have some data X on which I want to do the following: Train two models; SVM and Logistic Regression Use a stacking classifier based on the models from (1) ...
CutePoison's user avatar
0 votes
1 answer
90 views

Does double cross-validation make sense?

We do the following: split data_all into K folds, each consisting of data_train_k and ...
Brzoskwinia's user avatar

1
2 3 4 5
13