Questions tagged [cross-validation]
Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.
639
questions
0
votes
3
answers
58
views
Why the one validation score is lower than the other sections of cross validation
I was working on RandomForestClassifier and doing hyperparameter tuning. But something caught my attention. I always get a lower validation value in the 2nd part of Cross Validation. Here is the code:
...
0
votes
0
answers
10
views
Stable test in online time series forecasting problem
I have a Time Series Forecasting problem. You can think of it as predicting the daily closing prices of Apple stocks. My data is divided into 4-day segments, and the forecasting is based on predicting ...
- 63
0
votes
1
answer
49
views
Is Repeated K-Fold Cross Validation Enough to Evaluate a Machine Learning Model?
I am training models with a small dataset (around 800 observations) and I am using Repeated K-Fold cross validation to evaluate the models.
Initially, i am using the same cross validation for ...
0
votes
0
answers
23
views
XGBoost Classifier Evaluation Confusion on New Dataset Despite High Cross-Validation Scores
I have built an XGBoost classifier model with 90 features, trained on a dataset containing 760k samples. I took great care to separate the labels from the features in both the training and testing ...
- 1
0
votes
1
answer
21
views
How to pass a Dataframe as train dataframe and another dataframe as Validation to GridSearchCV
I'm a programmer who tries to find he's way into ML world. so the Question might be basic.
i have data from years 2010-2019. Now i'm trying to test different parameters on gradient boosting regression ...
- 101
1
vote
1
answer
65
views
Trying to understand Nested k-fold CV in a paper
I fully understand the data partition in a nested k-fold CV. But reading this:
Within each outer fold, the best performing model was selected based on mean root mean squared error (RMSE) over the
...
-1
votes
1
answer
93
views
ARMA model using different train and test/validation datasets
In sklearn I am used to having distinct train and test datasets. In other words, I train a model's parameters on the features from a training set, and then apply ...
- 99
0
votes
0
answers
39
views
Classification Threshold Optimization after GridSearchCV
In my machine learning problem I am using a CNN to classify images.
Since my dataset is imbalanced I want to perform classification probability threshold tuning so I can find the optimal balance ...
0
votes
2
answers
33
views
Optimal Data Split
I have a multiclass problem (3 classes) that looks to predict if someone will buy a product, neutral or not. I have initial features of in-app activity data such as likes, share, bookmark, share, ...
0
votes
0
answers
10
views
Test accuracy plateaus when increasing max_depth -> inf
I've built a Random Forest model that classifies into four categories based on around 10 input features.
To test the accuracy, I performed 5-fold stratified cross validation using the ...
0
votes
1
answer
29
views
Cross validation and train_test_split
I am building a class that follows the workflow:
Model Selection and Fitting
The class accepts a list of models and their respective hyperparameter grids. It then performs a standard fitting process ...
0
votes
3
answers
115
views
For cross validation should I use training set, or whole dataset?
I'm new to data science and I have a problem understanding what dataset to use when using cross validation for model evaluation.
Let's say I have two models: LogisticRegression and ...
0
votes
1
answer
24
views
Challenges in Predicting Molecule Activity
I want to share a concern I have. I want to obtain a machine learning model that can predict whether a molecule exhibits biological activity. For this purpose, I have a set of molecules that do ...
0
votes
0
answers
35
views
ML model to predict CPU utilization of a server given x amount of tasks
I have comprehensive data points of what the CPU utilization of a server is when x amount of jobs are running, let's say the server is using 40% CPU util time=x and there are 4 jobs running.
The ...
0
votes
0
answers
48
views
Build a model with cross-validation on entire dataset to learn insights?
Goal : Use XGBoost regression to learn insights from data. Prediction or forecasting not needed.
Hypothesis : If the model fits the entire dataset well, it can maybe capture its "physics" in ...
0
votes
0
answers
11
views
Can we apply cross validation bias correction to explanatory variable selection as well?
As shown in the paper below, several methods have been suggested for bias correction for cross validation. For example, Tibshirani-Tibshirani method and BBC-CV method. These are known to significantly ...
- 1
1
vote
1
answer
53
views
integration of Feature Selection in Pipeline
I have noticed integrating feature selection in a pipeline alters results.
Pipeline 1 gives slightly different results with pipeline 2. Why should this be so?
Pipeline 2
...
- 113
0
votes
1
answer
76
views
Can I use GridSearchCV.best_score_ for evaluation of model performance?
Scikit-learn page on Grid Search says:
Model selection by evaluating various parameter settings can be seen
as a way to use the labeled data to “train” the parameters of the
grid.
When evaluating the ...
- 103
1
vote
0
answers
93
views
What is the best way to combine cross-validation and bootstrapping for one application?
We intend to model data with non-parametric covariate splines and we would like to understand the uncertainty of the parameter estimates/response estimates.
Currently, we use cross-validation to model ...
- 111
0
votes
0
answers
9
views
Validation error less than training error
entire project link: Github Repository
In a classification task using Neural Network, I computed the fraction of misclassification as an error. And I am getting a validation error less than the ...
0
votes
1
answer
41
views
Why does my custom rmsle_loss produce negative scores during cross-validation?
I have a pipeline and scorer that produces some unexpected behaviors during cross-validation. During cross validation the pipeline produces all negative scores, but from the definition from rmsle_loss ...
- 5
0
votes
0
answers
30
views
Surprise NMF object is not callable
I am building a recommender system using the Sushi Preference Dataset and the NMF (Non-negative Matrix Factorization) model. I am implementing the same using the Surprise library. I want to use ...
1
vote
2
answers
41
views
which hyperparameters are returned as best in cross validation?
The description on the RandomizedSearchCV says this about best hyperparameters :
"Estimator that was chosen by the search, i.e. estimator which gave highest ...
- 31
0
votes
1
answer
24
views
error when using KFold() and roc_auc metric
why cross_val_score(pipe,X,y,scoring="roc_auc",cv=StratifiedKFold()) works just fine and when using KFold() like ...
- 51
0
votes
0
answers
26
views
Unsupervised clustering approach validated using internal data
I have used mclust package in R software for unsupervised clustering approach and choosing the clustering result according to the minimum BIC value. Can I use the cross validation method and calculate ...
- 1
0
votes
0
answers
22
views
Is subsequent cross validation on the same dataset biased?
I am training ML regression models to predict financial returns in a high frequency trading environment.
I have 1 time-series of intraday data for 40 years for 1 individual security at the moment.
I ...
- 1
0
votes
1
answer
39
views
Is this the best method for comparing different approaches nd selecting the best model in machine learning?
My objective is to experiment with various approaches for different algorithms, identify the best approach for each algorithm, and subsequently determine the best overall algorithm from among these ...
1
vote
1
answer
51
views
scikit-learn cross_val_score randomness
Does cross_val_score in scikit-learn split the data consistently or randomly? I noticed that cross_val_score lacks a random_state parameter, but the documentation mentions stratified k-fold cross-...
- 51
0
votes
0
answers
12
views
Fairness post-processing changing threshold and margin
I have a doubt, I want to use fairness methods and I see the tutorial they split test into test and validation and use validation to determine the best threshold and margin for predictions. But I ...
0
votes
0
answers
79
views
Visualize Catboost and XGBoost training process + Cross Validation
I want to optimize Catboost and XGBoost models and visualize this process such that:
Use 3-fold cross-validation
Use my own pre-processing pipeline (Missing value imputation, over- or undersampling)
...
- 61
0
votes
3
answers
143
views
How to properly do a k-fold cross validation?
I am trying to solve binary classification problem using deep neural networks. I want to compare different approaches (model architectures) and I have no hyperparameters which I want to tune. So my ...
- 13
0
votes
0
answers
50
views
Walk forward cross-validation with Optuna and deepar in pytorch forecasting
I want to perform 3 splits walk forward cross validation with expanding training set for the deepar model from the pytorch forecasting framework. When I do walk forward validation, I also want to do ...
- 101
0
votes
1
answer
27
views
Flow of machine learning model including code
I'm towards the completion of my first data science project that will go into my GitHub portfolio.
I'll be happy for some clarification regarding the machine learning models section:
I got a little ...
- 15
0
votes
1
answer
46
views
What is the point of final test set in K-fold cross-validation?
I am carrying out logistic regression for my binary classification problem, and I have validated the model with kfold cross-validation (k=10). I don't understand why I need to have a final test set, ...
0
votes
0
answers
29
views
Should I use mean, standard deviation, or coefficient of variation in cross-validation?
I'm using cross-validation and calculating the AUCs, and then calculating the mean, the standard deviation(SD), and taking the same standard deviation and dividing it by the same mean to calculate the ...
- 89
0
votes
0
answers
31
views
Is this a valid cross validation approach to choose hyperparameters and get a good estimate of model performace?
I am using lightGBM on time series data. I first split my data set into 10% folds. The last fold is used as a test set.
For each choice of hyperparameters I first use 6 folds to train, then predict on ...
0
votes
0
answers
17
views
Can I firstly use LazyPredict and RandomizedSearchCV later?
If I have a training a test set, X_train and X_test, can I firstly use LazyPredict to find the best model and later tuning that model's parameters using RandomizedSearchCV?
My doubt is that the ...
1
vote
2
answers
239
views
RandomizedSearchcv(n_iter=10) doesnt stop after training 10 models
I am using RandomizedSearchcv for hyperparameter optimization. When I run the model, it shows the scores for each model training. The problem is, it trains way more than 10 models when in fact I ...
- 31
0
votes
1
answer
62
views
How do I know If my regression model is underfitting?
How do we evaluate the performance of a regression model with a certain RMSE given that a domain knowledge performance metric is not present?
Maybe MAPE is one way of comparing the performance of my ...
- 31
0
votes
0
answers
34
views
Common cross-validation code: why does it work?
The following Python code is common practice when creating a folds column for multi-label stratified k-fold cross-validation:
...
- 101
0
votes
1
answer
58
views
Fit multiple models e.g classifiers -> stacking -> calibration without data-leak or getting too many datasets
I have some data X on which I want to do the following:
Train two models; SVM and Logistic Regression
Use a stacking classifier based on the models from (1)
...
- 460
0
votes
1
answer
53
views
Does double cross-validation make sense?
We do the following:
split data_all into K folds, each consisting of data_train_k and ...
- 103
0
votes
0
answers
51
views
How to properly perform K-fold cross validation with train, eval and test sets while building NN model?
Intro
I am training simple neural network and want to properly evaluate my model. It is not entirely clear to me, which dataset should I divide into folds in K-fold CV working with train, eval and ...
- 103
0
votes
1
answer
129
views
Stratifed time series split with the same imbalance ratio
I am recently working on an imbalanced binary classification problem where the data is time ordered. I would like to validate my model using training/validation splits that have the same imbalance ...
0
votes
0
answers
14
views
Why does cross validation and hyperparameter tuning work?
To my understanding, optimizing a model with k-fold cross validation and hyperparameter tuning are tools to be used mostly for small datasets to really make the most out of very limited/expensive data,...
- 1
0
votes
1
answer
44
views
Is it cheating to use normal KFold for data that is collected over time?
I am in doubt when to use strict time-series cross validation and when to use kfold. I have the following situation, which, I believe, is an edge-case between time series and normal data:
I have a ...
- 113
1
vote
1
answer
67
views
Which of 2 options is better practice for model optimization: 1) Nested CV wrongly averaging inner CV scores. 2) Two successive CVs on X_all. Altrntv?
Goal: Compare preprocessing methods, models, and hyperparameters without leaking into the final generalization estimate, applying cross-validation (cv), i.e. NOT applying any fixed train/test splits.
...
- 11
0
votes
0
answers
64
views
Objective function in optuna seems to return at random points
I am trying to use a K-fold cross-validation within the optuna objective function.
Unfortunately, the output of the objective function seems to pop out at random places, not at the end of the cross-...
0
votes
1
answer
25
views
What do I make of all classification scores being equal to 1?
I've built an XGBoost classifier on a dataset that has 51 columns and a 1000 rows with following code:
...
- 103
0
votes
1
answer
227
views
Tuned model has higher CV accuracy, but a lower test accuracy. Should I use the tuned or untuned model?
I am working on a classification problem using Sci Kit Learn and am confused on how to properly tune hyper parameters to get the "best" model.
Before any tuning, my logistic regression ...
