Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

Filter by
Sorted by
Tagged with
0
votes
1answer
259 views

Error while using lightGBM's cv() function for a regression problem

I am trying to use lightGBM's cv() function for tuning my model for a regression problem. My main model is ...
4
votes
1answer
466 views

Cross-validation for model comparison: use the same folds?

Let's say we have model M1 and model M2 that we want to compare. When we do 5-fold (say) cross validation, would the correct method to be to partition the data into F1, F2, F3, F4, and F5 and then run ...
2
votes
0answers
33 views

Cross Validation in Neural Networks

I am training a neural network and doing 10-fold cross validation to measure performance. I have read lots of documentation and forums telling that the set of weights that should be saved or ...
0
votes
1answer
21 views

Is it necessary to use stratified sampling if I am using SMOTE already?

I have already applied SMOTE to my imbalanced dataset with more than 300K observations. Does it still make sense to use stratified K-fold cross validation rather than simply ordinary K-fold cross ...
1
vote
0answers
13 views

How should you preprocess the data before K-fold validation?

I often see Kaggle notebook authors preprocessing the entire training data prior to splitting it for K-fold validation, but does this have a risk of leaking information into the validation set each ...
0
votes
0answers
62 views

Cross-validation in CNN

I am trying to implement deep learning for image recognition, but I am still confused with cross-validation. Let's say I will use about 100.000-1.000.000 images in total for a binary classification ...
1
vote
0answers
24 views

How to implement kfold and cv into Hybrid feature selection and evaluate the classification model performance?

I have been working on a Hybrid feature selection combined with hyperopt package for hyperparameter tuning and I am thinking about evaluating the performance of several model classifiers. I looked ...
0
votes
2answers
33 views

aggregation of feature importance

I have more of a conceptual question I was hoping to get some feedback on. I am trying to run a boosted regression ML model to identify a subset of important predictors for some clinical condition. ...
1
vote
2answers
35 views

Understanding Sklearns learning_curve

I have been using sklearns learning_curve , and there are a few questions I have that are not answered by the documentation(see also here and here), as well as questions that are raised by the ...
0
votes
1answer
33 views

Query regarding surprising spike in accuracy of ML model

I implemented all the major ML models (Logistic Regression, Naive Bayes, SVM, KNN, Decision Tree, Random Forest, Ada Boost & XGBoost) on my dataset. My stratified cross-validation scores are ...
0
votes
1answer
54 views

Learning curves

I am working on a multiclass classification problem. I want to know whether my model is overfitting or underfitting. I am learning how to plot learning curves and have 4 doubts. 1.) Is the ordering of ...
0
votes
3answers
58 views

Bias and variance in the model o in the predictions?

This topic confuses me. In the literature or articles, when talking about bias and variance in automatic learning, specifically in cross-validation, do they refer to the high bias (underfitting) and ...
0
votes
1answer
561 views

splitting into train test by train_test_split of float values?

How to split into train test by train_test_split of float values ? I used LabelEncoder but I have about 300K lines and when I used the cross_val I saw ...
0
votes
1answer
25 views

Comparing accuracies of Grid Search CV & Randomized Search CV with K-Fold Cross Validation?

Are Grid Search CV & Randomized Search CV always/necessarily supposed to give more accurate results after hyperparameter tuning as compared to K-Fold Cross Validation?
0
votes
0answers
43 views

K-fold cross-validation with XGBoost

I'm little bit confused concerning using k-fold CV along with XGBoost. So I had a multi-class classification problem and I decided to try something new and work with XGBoost. I've read in some article ...
0
votes
1answer
46 views

How to use Cross Validation data set?

I am new to data science, and the dataset I am working on is divided into train set, test set, and validation set. However, till now I was splitting the data with ...
0
votes
1answer
29 views

Steps of multiclass classification problem

So this question is more theoretical, than a practical one. I got a dataframe with 4 classes of cars' body types (e.g. sedan, hatchback, etc.) and different characteristics (doors, seats, maximum ...
2
votes
2answers
108 views

Removing the outliers improved my models, is what I did good or bad?

I used cross validation on my data (11000 rows) with maximum salary of 10000 and after some cleaning I got to rmse=70. Then I tried to remove the outliers 10 times just to try things now I have 9000 ...
1
vote
1answer
41 views

Splitting the dataset manually for k-Fold Cross-Validation

I manually divided the dataset into three sets: train, test, and validation. Each set includes several folders, one for each patient. Each patient has many images from a different point of view. As a ...
1
vote
1answer
61 views

What to do about non-responses to demographic survey questions?

Can anyone point me to some think pieces on what to do with census type data where at least some of the people surveyed do not self-identify a race/ethnicity, gender, or other demographic data? In ...
0
votes
1answer
40 views

SKLearn - Understanding Discrepancy Between LogisticRegressionCV classification_report and scores_

Cross-posting from Stack Overflow: I'm running into a weird situation where my sklearn LogisticRegressionCV model is apparently getting 100% accuracy (the lack of ...
1
vote
1answer
321 views

Confusion in applying k-fold Cross Validation to dataset

I have a dataset which is already divided into 10 folds with each fold having training, validation and test sets. I'm not able to understand how to apply 10-fold cross validation on this dataset. In ...
0
votes
2answers
58 views

I am attempting to implement k-folds cross validation in python3. What is the best way to implement this? Is it preferable to use Pandas or Numpy? [closed]

I am attempting to create a script to implement cross validation in data. However, the splits cannot randomly take any records, so the training and testing can be done on equal data splits for each ...
2
votes
2answers
67 views

Does high accuracy metrics with small (but equally sampled) dataset means a good model?

I have been training my CNN with 200 images per class for a classification problem. There problem is a binary classification one. And with the amount of test data ( 25 per class) I am getting good ...
2
votes
1answer
697 views

How do I perform Leave One Out Cross Validation For Top n Recommendation Sytems?

I am new in making recommendation systems . I am using the surpriselib library to evaluate my recommendations. All the Accuracy Metrics are well supported in this library. But I also want to compute ...
6
votes
2answers
189 views

ground truth fit is worse than cross validated fit on noisy data?

I am having these weird results when playing around with cross-validation that I would greatly appreciate having any comments. Briefly, I have a lower mean squared error (MSE) when doing regression (...
7
votes
1answer
947 views

Why you shouldn't upsample before cross validation

I have an imbalanced dataset and I am trying different methods to address the data imbalance. I found this article that explains the correct way to cross-validate when oversampling data using SMOTE ...
1
vote
2answers
42 views

How to get best data split from cross validation

I have trained a Random forest regressor which is giving me a rmse score of 70.72. But when I tried the same model in cross_val_score with a cv of 10, it gave me an array that looks something like ...
4
votes
1answer
66 views

What parameters to use when normalising training, validation, and testing data?

I know a similar post was made here, but I wanted to ask some follow up questions. I am conducting a cross-validation search to find values of a set of hyper-parameters and need to normalise the data. ...
1
vote
1answer
52 views

Cross-validation split for modelling data with timeseries behavior

Background: I have a dataset that is generated every month (it is similar with card data that contains card demography and transactions every month and new accounts can be added in the middle of data ...
2
votes
3answers
3k views

how to prepare data for cross validation in mnist dataset?

How to use k-fold cross validation for MNIST dataset? I read article documentation on sci-kit learn ,in that example they used the whole iris dataset for cross validation. ...
0
votes
1answer
50 views

Compare cross validation values of Bernoulli NB and Multinomial NB

I'm testing the Multinomial NB and Bernoulli NB on my dataset and I'm using the cross validation score to better understand which of the two algorithms work better. This is the first classifier: ...
2
votes
1answer
18 views

leave one pair out cross validation

I am trying to train and validate my datasets which contains 17 datasets. I have divided them as 15 for training and 2 for validation. In the process, I train on 15 datasets and use the generated ...
1
vote
1answer
23 views

Can I apply different hyper-parameters for different sliding time windows?

Question Can I apply different hyper-parameters for different training sets? I can see the point of using the shared parameters but I cannot see the point of using shared hyper-parameters. The ...
1
vote
1answer
530 views

Decision Trees change result at every run, how can I trust of my results?

Given a database, I split the data in train and test. I want to use a decision-tree classifier (sklearn) for a binary classification problem. Considering I already found the best parameters for my ...
5
votes
2answers
3k views

python xgboost DMatrix - get feature values or convert to np.array

I'm trying to create a custom evaluation metric (feval) function for xgboost.cv. It should process some of the training features, however I can't find a way to extract features from ...
1
vote
1answer
26 views

How can I use the K-fold cross validation for one-class classifier? [closed]

I train a one-class classifier by Class X. In the testing stage, I use Class X and Class Y for validation. I want to compute the F-score metric of Class Y. How can I use the K-fold cross-validation ...
2
votes
1answer
581 views

Optimizing decision threshold on model with oversampled/imbalanced data

I'm working on developing a model with a highly imbalanced dataset (0.7% Minority class). To remedy the imbalance, I was going to oversample using algorithms from imbalanced-learn library. I had a ...
4
votes
1answer
226 views

Why does CV yield lower score?

My training accuracy was better than my test accuracy, hence I thought my model was over-fitted and tried Cross-validation. The model further degraded. Is that my input data need to be sanitised ...
1
vote
2answers
43 views

Why cant we further tune/change the model after evaluating on the test set?

Every thread on stackexchange that I've found says that you can only use the test set once and thats it. So for instance, if you used a linear regression model and got poor results on the test set, ...
0
votes
0answers
29 views

Building a custom scoring function to find mean time-dependent AUC

I’m working on a survival analysis to predict 1-year mortality. I’m trying to build a custom score function that maximizes mean time-dependent AUC. Here is a description of the time-dependent AUC ...
3
votes
1answer
60 views

Restrictions on my skewed validation data

I have a severely skewed data sets consisting of 20 something classes where the smallest class contains on the order of 1000 samples and the largest several millions. Regarding the validation data, ...
0
votes
1answer
143 views

Splitting large multi class dataset using leave one out scheme into train and test

I am doing some supervised learning using neural networks, and i have a Targets array containing 1906 samples, which contain 664 unique values. min. count of each unique value==2, by design. Is there ...
0
votes
0answers
15 views

Qaulity assurance framework for ML/AI

I'd like to hear how organisations are applying a quality framework to ML/AI work. I'm struggling to find good content or good practice on this. By saying quality frameworks, for example, if I was ...
0
votes
1answer
44 views

New to keras neural network and k-fold cross validation

I'm new to learning neural networks and I found an example online to test accuracy with k-fold cross validation. The example is for binary data but I want to test MAE or RMSE (I guess?) for my ...
0
votes
1answer
184 views

Correctly evaluate model with oversampling and cross-validation

I'm dealing with a classic case of dataset with binary imbalanced target (event 3%, non event 97%). My idea is to apply some sort of sampling (over/under, SMOTE etc.) to address the issue. As I see, ...
0
votes
0answers
16 views

Training and validation accuracy stagnating after a few epochs for text embeddings

I have text embeddings (768 dimensional vectors). I tried to build a feed forward neural network on classify the text into two classes. The network I used. ...
1
vote
2answers
718 views

Using keras with sklearn: apply class_weight with cross_val_score

I have a highly imbalanced dataset (± 5% positive instances), for which I am training binary classifiers. I am using nested 5-fold cross-validation with grid search for hyperparameter tuning. I want ...
0
votes
0answers
25 views

Cross-validation for Random forest

I want to use cross-validation method for Random forest and compare the performance with other algorithms in the same condition. I heard from somebody that Random forest does not need cross-validation....
3
votes
1answer
2k views

K-fold cross validation of scikit-learn with confusion matrix of Keras

I intend to display a confusion matrix using Keras while K-fold of scikit-learn. My code using Keras is: ...

1
2 3 4 5
11