Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

2
votes
3answers
65 views

Cross validation Vs. Train Validate Test

I have a doubt regarding the cross validation approach and train-validation-test approach. I was told that I can split a dataset into 3 parts: Train: we train the model. Validation: we validate and ...
0
votes
1answer
22 views

K-Fold and Random Subsampling (RSS) Dataset generation?

Let say if I have a large dataset of 300k instances with 200 features, I want to reduce its size. Can I apply K-Fold technique to the 200 features then the trimmed dataset are applied with RSS to trim ...
0
votes
1answer
13 views

Applying Hold-out and CV technique

I have a methodology question: are hold-out and CV generalization-optimization techniques mutually exclusive? It gets really confusing to me at times, because in the most recent project I have been ...
0
votes
1answer
16 views

Am I using GridSearch correctly or do I need to use all data for cross validation?

I'm working with a dataset that has 400 observations, 34 features and quite a few outliers, some of them extreme. Given the nature of my data, these need to be in the model. I started by doing a 75-...
2
votes
1answer
38 views

Cross-validation for model comparison: use the same folds?

Let's say we have model M1 and model M2 that we want to compare. When we do 5-fold (say) cross validation, would the correct method to be to partition the data into F1, F2, F3, F4, and F5 and then run ...
2
votes
1answer
23 views

Help with understanding cross-validation

My understanding of cross-validation is that we divide our data set into parts 1-k, then use part 1 as a validation set and parts 2-k as a training set, then use part 2 as a validation set and the ...
3
votes
1answer
63 views

Model comparison with CV using standard error

Discovering the ML world with sklearn, I'm testing a large panel of models onto my dataset. This is for learning purpose but also for work so I want the final model ...
2
votes
1answer
137 views

Which model to chose based on learning curve

I trained my model using different regression techniques, and I'm not sure which model to choose based on the learning curve. 1) Should I choose Lasso, since train and CV converge at the end 2) ...
2
votes
1answer
35 views

Hyper parameters tuning XGBClassifier

I am working on a highly imbalanced dataset for a competition. The training data shape is : (166573, 14) ...
3
votes
1answer
64 views

Can we use k fold Cross Validation without any extra (excluded) Test Set?

I have seen this in two Papers: The authors use 10 fold cross validation, and then present the results from this validation or even odder the results from the best Fold as their modelling Result. ...
0
votes
0answers
22 views

Error while using lightGBM's cv() function for a regression problem

I am trying to use lightGBM's cv() function for tuning my model for a regression problem. My main model is ...
2
votes
1answer
129 views

XGBRegressor hyperparameter optimization using xgb cv function

I am trying to optimize hyper parameters of XGBRegressor using xgb's cv function and bayesian optimization (using hyperopt package). Here is the piece of code I am using for the cv part. ...
0
votes
1answer
35 views

Is the role of the validation set in a deep learning network is only for Early Stopping?

In the "deep learning crash course" given by Leo Isikdogan in lecture 4 https://www.youtube.com/watch?v=ms-Ooh9mjiE&list=PLWKotBjTDoLj3rXBL-nEIPRN9V3a9Cx07&index=4 Overfitting, Underfitting, ...
0
votes
0answers
4 views

Getting parameters of the best model with crossvalidation in with SparkMLLib

I am having trouble accessing the parameters of estimators of model in SparkMLlib. More precisely my problem is: I have a logistic regression model for which I want to find the best regularization ...
2
votes
0answers
39 views

Is AUC a good metric for evaluating the performance of a multi-class classification?

Considering the definition of AUC (Area Under Curve), is that a reliable performance metric for a multi-class (30-40 classes) classification problem?
3
votes
0answers
96 views

Why does CV yield lower score?

My training accuracy was better than my test accuracy, hence I thought my model was over-fitted and tried Cross-validation. The model further degraded. Is that my input data need to be sanitised ...
1
vote
1answer
65 views

How to determine number of leaves in decision tree analysis?

Would be grateful if some expert on the forum can help me understand how to decide optimum number of leaves in a decision tree analysis. I am using SAS and if I supply leaves=6 in my model then miss-...
1
vote
0answers
55 views

Plotting ROC curve in cross validation using Matlab perfcurve

I have the following code for a binary classifying using SVM, and 10 cross-validation, ...
7
votes
2answers
555 views

Validation vs. test vs. training accuracy. Which one should I compare for claiming overfit?

I have read on the several answers here and on the Internet that cross-validation helps to indicate that if the model will generalize well or not and about overfitting. But I am confused that which ...
0
votes
1answer
14 views

Choose CNN architecture first, then optimize parameters - validation vs test performance to pick architecture?

I am doing a few experiments on medical data. I am about to transfer learn the pretrained networks for my problem. Firstly, I have to pick a network architecture. Secondly, I would like to optimize ...
0
votes
1answer
31 views

Evaluating the test set

Please find attached a part of the code which explains what I'm trying to do. Essentially I'm trying to predict the sales of supermarket stores. Im using RandomForestRegressor for this and have ...
0
votes
0answers
40 views

What to do when Kfold is not enough?

I have a dataset made of roughly 100 time-series and my final goal is to obtain a classification of each point (detection problem). To do so I have labels so I decided to use an XGB model to perform ...
-1
votes
1answer
324 views

k-fold cross validation in keras for regression using sklearn [closed]

I am using a wrapper to use sklearn k-fold cross-validation with keras for a regression problem with ANN. but the accuracies i get look very weird. It has worked fine for a classification problem. I ...
1
vote
1answer
15 views

Change rate of cross validation data, after training

Say we have N of labeled data, and we need to take some part for the cross validation (we will skip test part for this case). We ...
3
votes
1answer
115 views

Overfitting - how to detect it and reduce it?

I have a side project where I am doing credit scoring using R (sample size around 16k for train data and 4k for test data, and also another two 20k data batches for out-of-time validation) with ...
1
vote
1answer
44 views

80-20 better than full dataset for LightGBM

Recently I have been using LightGBM as regressor in order to predict, on a dataset of 20 thousand observations and 40 variables. I have two modes, 1) Production and 2) Testing. The first one just ...
0
votes
0answers
46 views

How can I recognise if I can improve a random forest model by adding features

I want to tune a random forest model with caret package. I'm tuning it with cross-validation to prevent overfitting and resulted cross-validation accuracy is very ...
0
votes
0answers
27 views

Cross-validation and out-of-bag bootstrap applications

I have a question regarding steps on which a specific resample method should be used in general. As far as I know: out-of-bag bootstrap is the resample method with replacement, which has lower ...
2
votes
1answer
52 views

Should I oversample my validation data to get better F1 score and PRC?

I am currently working with a dataset that is imbalanced, about 30k rows * 14 features (just for you know), and 99.5% of the data is labeled 0. Since the model is strongly imbalanced I decided to use ...
1
vote
0answers
15 views

Performance diagnostics in mxnet gluon (e.g. plotting training vs validation loss over time)?

Tensorflow has tensorboard, is there any recommended way to plot classification error/loss over time in mxnet?
2
votes
1answer
39 views

Is splitting the data set into train and validation applicable in unsupervised learning?

I am having a tough time implementing all the steps of setting up support vector machine (SVM) for unsupervised learning. My data set is labelled but for educational purposes I am learning ...
1
vote
1answer
95 views

Target encoding with cross validation

I am trying to understand this way of target (mean/impact/likelihood) encoding using (two-level) cross validation. It's taking mean value of y. But not plain mean, but in cross-validation within ...
3
votes
1answer
190 views

PCA, SMOTE and cross validation- how to combine them together?

I was reading a lot recently about PCA and cross validation and it seems that the majority call it malpractice to do PCA before cross validation. I would also like to perform SMOTE, but there is a ...
3
votes
1answer
202 views
0
votes
0answers
47 views

Validation curve

I'm learning about data science and I've been checking several tutorials. Now I'm trying some validation curves on the problem sample I'm resolving and I'm having some troubles with it. This is the ...
0
votes
0answers
31 views

Validition score while training lower than on final model with xgboost

I have 3 three classes, but my metric is auc, so I have customer eval metric: ...
1
vote
1answer
32 views
2
votes
1answer
290 views

K-fold cross validation of scikit-learn with confusion matrix of Keras

I intend to display confusion matrix using Keras while K-fold of scikit-learn. My code using Keras is: ...
1
vote
0answers
24 views

Regression model Giving the same prediction for all new inputs until i load the model again

I have build a regression model that has some decent accuracy measures. I have pickled it and loading it another project. However it is producing the same predictions every time when i pass new ...
5
votes
2answers
35 views

Does a precision score increasing with a higher number of folds mean the model will improve with more data?

I have been working on a pretty simple text classifying module (tfidf + Random Forest). My manager insisted on using a simple .7/.3 split rather than doing cross validation, then was adamant about ...
1
vote
1answer
151 views

Using keras with sklearn: apply class_weight with cross_val_score

I have a highly imbalanced dataset (± 5% positive instances), for which I am training binary classifiers. I am using nested 5-fold cross-validation with grid search for hyperparameter tuning. I want ...
5
votes
1answer
257 views

Cross validation for highly imbalanced data with undersampling

In my problem, I am dealing with a highly imbalanced data set, say for every positive class there are 10000 negative one. A normal starting method to train a model is to undersample the data. In this ...
1
vote
0answers
25 views

how to label a tain_data? [closed]

I have one assignment that I have four files 1) train_data.csv: The training file contains two fields (text, id). 2) train_label.csv: The label file contains two fields (id, label). 3) test_data.csv: ...
1
vote
0answers
62 views

Improve model performance on unseen data

NOTE: This question was first posted on a different SO forum but I received suggestions to move it here This is a follow-up question to a question I had previously posted on this forum We conducted ...
0
votes
0answers
13 views

Does sklearn's gridsearchCV use the same cross validation train/test splits for evaluating each hyperparameter combination?

I couldn't find the answer on any forum in the interwebs so I hunted down the answer myself. Yes; the same train/test splits are used for each parameter combination. Here Is the relevant sklearn ...
0
votes
0answers
13 views

Name for non-exhaustive cross validation technique similar to leave-one-out but with test set size larger than one

Is there a name for a cross validation technique like k-fold that accepts the size of the fold instead of the amount of folds? If I understand it correctly, leave-p-out outputs all combinations, so ...
1
vote
0answers
18 views

Strangeness in validation loss between CPU vs GPU when training CNN

I've been training an implementation of Mask R-CNN and it was training very successfully on my CPU but I've just set up my GPU and it is giving some strange results when looking at my validation loss. ...
1
vote
1answer
26 views

Please help select an Algo based on Accuracy and Confusion Matrix

I am very new to Data Science would appreciate your advice big time. Got a task: predict if a trade will be profitable or not, based on a set of data. I have prepared, cleaned and tested data. ...