Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

Filter by
Sorted by
Tagged with
0 votes
3 answers
58 views

Why the one validation score is lower than the other sections of cross validation

I was working on RandomForestClassifier and doing hyperparameter tuning. But something caught my attention. I always get a lower validation value in the 2nd part of Cross Validation. Here is the code: ...
Emir Kutsal's user avatar
0 votes
0 answers
10 views

Stable test in online time series forecasting problem

I have a Time Series Forecasting problem. You can think of it as predicting the daily closing prices of Apple stocks. My data is divided into 4-day segments, and the forecasting is based on predicting ...
Angerato's user avatar
0 votes
1 answer
49 views

Is Repeated K-Fold Cross Validation Enough to Evaluate a Machine Learning Model?

I am training models with a small dataset (around 800 observations) and I am using Repeated K-Fold cross validation to evaluate the models. Initially, i am using the same cross validation for ...
codenoob1211's user avatar
0 votes
0 answers
23 views

XGBoost Classifier Evaluation Confusion on New Dataset Despite High Cross-Validation Scores

I have built an XGBoost classifier model with 90 features, trained on a dataset containing 760k samples. I took great care to separate the labels from the features in both the training and testing ...
oklen's user avatar
  • 1
0 votes
1 answer
21 views

How to pass a Dataframe as train dataframe and another dataframe as Validation to GridSearchCV

I'm a programmer who tries to find he's way into ML world. so the Question might be basic. i have data from years 2010-2019. Now i'm trying to test different parameters on gradient boosting regression ...
Mostafa Bouzari's user avatar
1 vote
1 answer
65 views

Trying to understand Nested k-fold CV in a paper

I fully understand the data partition in a nested k-fold CV. But reading this: Within each outer fold, the best performing model was selected based on mean root mean squared error (RMSE) over the ...
Amirhossein Rezaei's user avatar
-1 votes
1 answer
93 views

ARMA model using different train and test/validation datasets

In sklearn I am used to having distinct train and test datasets. In other words, I train a model's parameters on the features from a training set, and then apply ...
user3128's user avatar
0 votes
0 answers
39 views

Classification Threshold Optimization after GridSearchCV

In my machine learning problem I am using a CNN to classify images. Since my dataset is imbalanced I want to perform classification probability threshold tuning so I can find the optimal balance ...
Throwaway123's user avatar
0 votes
2 answers
33 views

Optimal Data Split

I have a multiclass problem (3 classes) that looks to predict if someone will buy a product, neutral or not. I have initial features of in-app activity data such as likes, share, bookmark, share, ...
Marc Atanante's user avatar
0 votes
0 answers
10 views

Test accuracy plateaus when increasing max_depth -> inf

I've built a Random Forest model that classifies into four categories based on around 10 input features. To test the accuracy, I performed 5-fold stratified cross validation using the ...
okjdlsksjdwi's user avatar
0 votes
1 answer
29 views

Cross validation and train_test_split

I am building a class that follows the workflow: Model Selection and Fitting The class accepts a list of models and their respective hyperparameter grids. It then performs a standard fitting process ...
Guilherme Raibolt's user avatar
0 votes
3 answers
115 views

For cross validation should I use training set, or whole dataset?

I'm new to data science and I have a problem understanding what dataset to use when using cross validation for model evaluation. Let's say I have two models: LogisticRegression and ...
Michał Jurzak's user avatar
0 votes
1 answer
24 views

Challenges in Predicting Molecule Activity

I want to share a concern I have. I want to obtain a machine learning model that can predict whether a molecule exhibits biological activity. For this purpose, I have a set of molecules that do ...
Yasser Hayek's user avatar
0 votes
0 answers
35 views

ML model to predict CPU utilization of a server given x amount of tasks

I have comprehensive data points of what the CPU utilization of a server is when x amount of jobs are running, let's say the server is using 40% CPU util time=x and there are 4 jobs running. The ...
cpuUtilServerHelp's user avatar
0 votes
0 answers
48 views

Build a model with cross-validation on entire dataset to learn insights?

Goal : Use XGBoost regression to learn insights from data. Prediction or forecasting not needed. Hypothesis : If the model fits the entire dataset well, it can maybe capture its "physics" in ...
cwanderroycbooks's user avatar
0 votes
0 answers
11 views

Can we apply cross validation bias correction to explanatory variable selection as well?

As shown in the paper below, several methods have been suggested for bias correction for cross validation. For example, Tibshirani-Tibshirani method and BBC-CV method. These are known to significantly ...
x H's user avatar
  • 1
1 vote
1 answer
53 views

integration of Feature Selection in Pipeline

I have noticed integrating feature selection in a pipeline alters results. Pipeline 1 gives slightly different results with pipeline 2. Why should this be so? Pipeline 2 ...
wwnde's user avatar
  • 113
0 votes
1 answer
76 views

Can I use GridSearchCV.best_score_ for evaluation of model performance?

Scikit-learn page on Grid Search says: Model selection by evaluating various parameter settings can be seen as a way to use the labeled data to “train” the parameters of the grid. When evaluating the ...
Charlie's user avatar
  • 103
1 vote
0 answers
93 views

What is the best way to combine cross-validation and bootstrapping for one application?

We intend to model data with non-parametric covariate splines and we would like to understand the uncertainty of the parameter estimates/response estimates. Currently, we use cross-validation to model ...
Stan Tendijck's user avatar
0 votes
0 answers
9 views

Validation error less than training error

entire project link: Github Repository In a classification task using Neural Network, I computed the fraction of misclassification as an error. And I am getting a validation error less than the ...
Dipen Pandit's user avatar
0 votes
1 answer
41 views

Why does my custom rmsle_loss produce negative scores during cross-validation?

I have a pipeline and scorer that produces some unexpected behaviors during cross-validation. During cross validation the pipeline produces all negative scores, but from the definition from rmsle_loss ...
Tim's user avatar
  • 5
0 votes
0 answers
30 views

Surprise NMF object is not callable

I am building a recommender system using the Sushi Preference Dataset and the NMF (Non-negative Matrix Factorization) model. I am implementing the same using the Surprise library. I want to use ...
Sumant Chopde's user avatar
1 vote
2 answers
41 views

which hyperparameters are returned as best in cross validation?

The description on the RandomizedSearchCV says this about best hyperparameters : "Estimator that was chosen by the search, i.e. estimator which gave highest ...
pppp_prs's user avatar
0 votes
1 answer
24 views

error when using KFold() and roc_auc metric

why cross_val_score(pipe,X,y,scoring="roc_auc",cv=StratifiedKFold()) works just fine and when using KFold() like ...
jxqbbb's user avatar
  • 51
0 votes
0 answers
26 views

Unsupervised clustering approach validated using internal data

I have used mclust package in R software for unsupervised clustering approach and choosing the clustering result according to the minimum BIC value. Can I use the cross validation method and calculate ...
許乃偉's user avatar
0 votes
0 answers
22 views

Is subsequent cross validation on the same dataset biased?

I am training ML regression models to predict financial returns in a high frequency trading environment. I have 1 time-series of intraday data for 40 years for 1 individual security at the moment. I ...
Cap_F's user avatar
  • 1
0 votes
1 answer
39 views

Is this the best method for comparing different approaches nd selecting the best model in machine learning?

My objective is to experiment with various approaches for different algorithms, identify the best approach for each algorithm, and subsequently determine the best overall algorithm from among these ...
Salah Amani's user avatar
1 vote
1 answer
51 views

scikit-learn cross_val_score randomness

Does cross_val_score in scikit-learn split the data consistently or randomly? I noticed that cross_val_score lacks a random_state parameter, but the documentation mentions stratified k-fold cross-...
jxqbbb's user avatar
  • 51
0 votes
0 answers
12 views

Fairness post-processing changing threshold and margin

I have a doubt, I want to use fairness methods and I see the tutorial they split test into test and validation and use validation to determine the best threshold and margin for predictions. But I ...
Esmeralda Ruiz Pujadas's user avatar
0 votes
0 answers
79 views

Visualize Catboost and XGBoost training process + Cross Validation

I want to optimize Catboost and XGBoost models and visualize this process such that: Use 3-fold cross-validation Use my own pre-processing pipeline (Missing value imputation, over- or undersampling) ...
Ars ML's user avatar
  • 61
0 votes
3 answers
143 views

How to properly do a k-fold cross validation?

I am trying to solve binary classification problem using deep neural networks. I want to compare different approaches (model architectures) and I have no hyperparameters which I want to tune. So my ...
dmasny's user avatar
  • 13
0 votes
0 answers
50 views

Walk forward cross-validation with Optuna and deepar in pytorch forecasting

I want to perform 3 splits walk forward cross validation with expanding training set for the deepar model from the pytorch forecasting framework. When I do walk forward validation, I also want to do ...
Jose_Peeterson's user avatar
0 votes
1 answer
27 views

Flow of machine learning model including code

I'm towards the completion of my first data science project that will go into my GitHub portfolio. I'll be happy for some clarification regarding the machine learning models section: I got a little ...
Sigal Cohen's user avatar
0 votes
1 answer
46 views

What is the point of final test set in K-fold cross-validation?

I am carrying out logistic regression for my binary classification problem, and I have validated the model with kfold cross-validation (k=10). I don't understand why I need to have a final test set, ...
Karoline Teller's user avatar
0 votes
0 answers
29 views

Should I use mean, standard deviation, or coefficient of variation in cross-validation?

I'm using cross-validation and calculating the AUCs, and then calculating the mean, the standard deviation(SD), and taking the same standard deviation and dividing it by the same mean to calculate the ...
FjkgB's user avatar
  • 89
0 votes
0 answers
31 views

Is this a valid cross validation approach to choose hyperparameters and get a good estimate of model performace?

I am using lightGBM on time series data. I first split my data set into 10% folds. The last fold is used as a test set. For each choice of hyperparameters I first use 6 folds to train, then predict on ...
user3376601's user avatar
0 votes
0 answers
17 views

Can I firstly use LazyPredict and RandomizedSearchCV later?

If I have a training a test set, X_train and X_test, can I firstly use LazyPredict to find the best model and later tuning that model's parameters using RandomizedSearchCV? My doubt is that the ...
Flavio Brienza's user avatar
1 vote
2 answers
239 views

RandomizedSearchcv(n_iter=10) doesnt stop after training 10 models

I am using RandomizedSearchcv for hyperparameter optimization. When I run the model, it shows the scores for each model training. The problem is, it trains way more than 10 models when in fact I ...
Mehmet Deniz's user avatar
0 votes
1 answer
62 views

How do I know If my regression model is underfitting?

How do we evaluate the performance of a regression model with a certain RMSE given that a domain knowledge performance metric is not present? Maybe MAPE is one way of comparing the performance of my ...
Mehmet Deniz's user avatar
0 votes
0 answers
34 views

Common cross-validation code: why does it work?

The following Python code is common practice when creating a folds column for multi-label stratified k-fold cross-validation: ...
tossimmar's user avatar
  • 101
0 votes
1 answer
58 views

Fit multiple models e.g classifiers -> stacking -> calibration without data-leak or getting too many datasets

I have some data X on which I want to do the following: Train two models; SVM and Logistic Regression Use a stacking classifier based on the models from (1) ...
CutePoison's user avatar
0 votes
1 answer
53 views

Does double cross-validation make sense?

We do the following: split data_all into K folds, each consisting of data_train_k and ...
Brzoskwinia's user avatar
0 votes
0 answers
51 views

How to properly perform K-fold cross validation with train, eval and test sets while building NN model?

Intro I am training simple neural network and want to properly evaluate my model. It is not entirely clear to me, which dataset should I divide into folds in K-fold CV working with train, eval and ...
Brzoskwinia's user avatar
0 votes
1 answer
129 views

Stratifed time series split with the same imbalance ratio

I am recently working on an imbalanced binary classification problem where the data is time ordered. I would like to validate my model using training/validation splits that have the same imbalance ...
Moataz Chouchen's user avatar
0 votes
0 answers
14 views

Why does cross validation and hyperparameter tuning work?

To my understanding, optimizing a model with k-fold cross validation and hyperparameter tuning are tools to be used mostly for small datasets to really make the most out of very limited/expensive data,...
frankL's user avatar
  • 1
0 votes
1 answer
44 views

Is it cheating to use normal KFold for data that is collected over time?

I am in doubt when to use strict time-series cross validation and when to use kfold. I have the following situation, which, I believe, is an edge-case between time series and normal data: I have a ...
Jasoba's user avatar
  • 113
1 vote
1 answer
67 views

Which of 2 options is better practice for model optimization: 1) Nested CV wrongly averaging inner CV scores. 2) Two successive CVs on X_all. Altrntv?

Goal: Compare preprocessing methods, models, and hyperparameters without leaking into the final generalization estimate, applying cross-validation (cv), i.e. NOT applying any fixed train/test splits. ...
le8rning's user avatar
0 votes
0 answers
64 views

Objective function in optuna seems to return at random points

I am trying to use a K-fold cross-validation within the optuna objective function. Unfortunately, the output of the objective function seems to pop out at random places, not at the end of the cross-...
Adrian Mureșan's user avatar
0 votes
1 answer
25 views

What do I make of all classification scores being equal to 1?

I've built an XGBoost classifier on a dataset that has 51 columns and a 1000 rows with following code: ...
Somanna's user avatar
  • 103
0 votes
1 answer
227 views

Tuned model has higher CV accuracy, but a lower test accuracy. Should I use the tuned or untuned model?

I am working on a classification problem using Sci Kit Learn and am confused on how to properly tune hyper parameters to get the "best" model. Before any tuning, my logistic regression ...
d0dg3r_k1d's user avatar

1
2 3 4 5
13