Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

7
votes
2answers
296 views

Validation vs. test vs. training accuracy. Which one should I compare for claiming overfit?

I have read on the several answers here and on the Internet that cross-validation helps to indicate that if the model will generalize well or not and about overfitting. But I am confused that which ...
0
votes
1answer
7 views

Choose CNN architecture first, then optimize parameters - validation vs test performance to pick architecture?

I am doing a few experiments on medical data. I am about to transfer learn the pretrained networks for my problem. Firstly, I have to pick a network architecture. Secondly, I would like to optimize ...
0
votes
1answer
29 views

Evaluating the test set

Please find attached a part of the code which explains what I'm trying to do. Essentially I'm trying to predict the sales of supermarket stores. Im using RandomForestRegressor for this and have ...
0
votes
0answers
38 views

What to do when Kfold is not enough?

I have a dataset made of roughly 100 time-series and my final goal is to obtain a classification of each point (detection problem). To do so I have labels so I decided to use an XGB model to perform ...
-1
votes
1answer
63 views

k-fold cross validation in keras for regression using sklearn [closed]

I am using a wrapper to use sklearn k-fold cross-validation with keras for a regression problem with ANN. but the accuracies i get look very weird. It has worked fine for a classification problem. I ...
1
vote
1answer
13 views

Change rate of cross validation data, after training

Say we have N of labeled data, and we need to take some part for the cross validation (we will skip test part for this case). We ...
0
votes
1answer
87 views

Overfitting - how to detect it and reduce it?

I have a side project where I am doing credit scoring using R (sample size around 16k for train data and 4k for test data, and also another two 20k data batches for out-of-time validation) with ...
1
vote
1answer
30 views

80-20 better than full dataset for LightGBM

Recently I have been using LightGBM as regressor in order to predict, on a dataset of 20 thousand observations and 40 variables. I have two modes, 1) Production and 2) Testing. The first one just ...
0
votes
0answers
21 views

How can I recognise if I can improve a random forest model by adding features

I want to tune a random forest model with caret package. I'm tuning it with cross-validation to prevent overfitting and resulted cross-validation accuracy is very ...
0
votes
0answers
20 views

Cross-validation and out-of-bag bootstrap applications

I have a question regarding steps on which a specific resample method should be used in general. As far as I know: out-of-bag bootstrap is the resample method with replacement, which has lower ...
2
votes
1answer
16 views

Should I oversample my validation data to get better F1 score and PRC?

I am currently working with a dataset that is imbalanced, about 30k rows * 14 features (just for you know), and 99.5% of the data is labeled 0. Since the model is strongly imbalanced I decided to use ...
1
vote
0answers
11 views

Performance diagnostics in mxnet gluon (e.g. plotting training vs validation loss over time)?

Tensorflow has tensorboard, is there any recommended way to plot classification error/loss over time in mxnet?
2
votes
1answer
32 views

Is splitting the data set into train and validation applicable in unsupervised learning?

I am having a tough time implementing all the steps of setting up support vector machine (SVM) for unsupervised learning. My data set is labelled but for educational purposes I am learning ...
0
votes
0answers
22 views

Target encoding with cross validation

I am trying to understand this way of target (mean/impact/likelihood) encoding using (two-level) cross validation. It's taking mean value of y. But not plain mean, but in cross-validation within ...
3
votes
1answer
75 views

PCA, SMOTE and cross validation- how to combine them together?

I was reading a lot recently about PCA and cross validation and it seems that the majority call it malpractice to do PCA before cross validation. I would also like to perform SMOTE, but there is a ...
2
votes
1answer
46 views
0
votes
0answers
25 views

Validation curve

I'm learning about data science and I've been checking several tutorials. Now I'm trying some validation curves on the problem sample I'm resolving and I'm having some troubles with it. This is the ...
0
votes
0answers
16 views

Validition score while training lower than on final model with xgboost

I have 3 three classes, but my metric is auc, so I have customer eval metric: ...
1
vote
1answer
26 views
2
votes
1answer
101 views

K-fold cross validation of scikit-learn with confusion matrix of Keras

I intend to display confusion matrix using Keras while K-fold of scikit-learn. My code using Keras is: ...
1
vote
0answers
21 views

Regression model Giving the same prediction for all new inputs until i load the model again

I have build a regression model that has some decent accuracy measures. I have pickled it and loading it another project. However it is producing the same predictions every time when i pass new ...
5
votes
2answers
31 views

Does a precision score increasing with a higher number of folds mean the model will improve with more data?

I have been working on a pretty simple text classifying module (tfidf + Random Forest). My manager insisted on using a simple .7/.3 split rather than doing cross validation, then was adamant about ...
0
votes
1answer
47 views

Using keras with sklearn: apply class_weight with cross_val_score

I have a highly imbalanced dataset (± 5% positive instances), for which I am training binary classifiers. I am using nested 5-fold cross-validation with grid search for hyperparameter tuning. I want ...
5
votes
1answer
144 views

Cross validation for highly imbalanced data with undersampling

In my problem, I am dealing with a highly imbalanced data set, say for every positive class there are 10000 negative one. A normal starting method to train a model is to undersample the data. In this ...
1
vote
0answers
25 views

how to label a tain_data? [closed]

I have one assignment that I have four files 1) train_data.csv: The training file contains two fields (text, id). 2) train_label.csv: The label file contains two fields (id, label). 3) test_data.csv: ...
1
vote
0answers
37 views

Improve model performance on unseen data

NOTE: This question was first posted on a different SO forum but I received suggestions to move it here This is a follow-up question to a question I had previously posted on this forum We conducted ...
0
votes
0answers
8 views

Does sklearn's gridsearchCV use the same cross validation train/test splits for evaluating each hyperparameter combination?

I couldn't find the answer on any forum in the interwebs so I hunted down the answer myself. Yes; the same train/test splits are used for each parameter combination. Here Is the relevant sklearn ...
0
votes
0answers
11 views

Name for non-exhaustive cross validation technique similar to leave-one-out but with test set size larger than one

Is there a name for a cross validation technique like k-fold that accepts the size of the fold instead of the amount of folds? If I understand it correctly, leave-p-out outputs all combinations, so ...
1
vote
0answers
16 views

Strangeness in validation loss between CPU vs GPU when training CNN

I've been training an implementation of Mask R-CNN and it was training very successfully on my CPU but I've just set up my GPU and it is giving some strange results when looking at my validation loss. ...
-1
votes
0answers
42 views

Are my Random Forest Classifier and Regressors overfitting?? I have CV and learning curves!

I seem to be getting great results from logistic regression with RFE and random forest feature importances in support, but there's been a suggestion of overfitting and when I run learning curves the ...
1
vote
1answer
26 views

Please help select an Algo based on Accuracy and Confusion Matrix

I am very new to Data Science would appreciate your advice big time. Got a task: predict if a trade will be profitable or not, based on a set of data. I have prepared, cleaned and tested data. ...
0
votes
1answer
20 views

Can accuracy become worse on the training set with more epochs?

I know that overfitting occurs when the accuracy on the training set improves but the accuracy on the validation set decrease. So, we must stop the training. I would like to know if this is a rule ...
0
votes
2answers
64 views

Validation accuracy is always close to training accuracy

I am trying to tune the hyperparameters of a LSTM I have to do time series forecasting. I have noticed that my validation accuracy is always very close to my training accuracy. I am not sure whether ...
1
vote
0answers
38 views

How do I perform Leave One Out Cross Validation For Top n Recommendation Sytems?

I am new in making recommendation systems . I am using the surpriselib library to evaluate my recommendations. All the Accuracy Metrics are well supported in this library. But I also want to compute ...
1
vote
1answer
203 views

cross validation for small dataset

I have a dataset of 39 medical MR images, and I have to build a model to classify the tumor type. so is it suitable to use k-fold cross validation for validating the model? if so, what would be the ...
1
vote
1answer
84 views

A few questions to understand a random forest blog [closed]

I'm trying to understand a nice blog on the trade-off between sensitivity versus specificity with the random forest and logistic regression models. I have a few questions: 1) The blog used a 10 fold ...
5
votes
4answers
180 views

Which is first ? Tuning the parameters or selecting the model

I've been reading about how we split our data into 3 parts; generally, we use the validation set to help us tune the parameters and the test set to have an unbiased estimate on how well does our model ...
1
vote
0answers
42 views

100% classification accuracy

I am trying to perform a multi-class classification where the network is trained to classify objects into 3 categories: cars, pedestrians and miscellaneous. I am using the KITTI Dataset for car ...
1
vote
1answer
58 views

How to apply Stacking cross validation for time-series data?

Normally stacking algorithm uses K-fold cross validation technique to predict oof validation that used for level 2 prediction. In case of time-series data (say stock movement prediction), K-fold ...
3
votes
1answer
145 views

Hyperparameter tuning for stacked models

I'm reading the following kaggle post for learning how to incorporate model stacking http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/ in ML models. The structure ...
1
vote
1answer
34 views

Is the ultimate challenge in ML simply computational power?

I am stuck on a theoretical roadblock in learning about machine learning, because I have not seen this explicitly addressed anywhere. In my studies, it seems as if Cross-validation (or some variant ...
0
votes
1answer
74 views

How can one use a validation set to reduce overfitting Naive Bayes?

What is the correct procedure for using a validation set to reduce overfitting? Say I split the data 80:10:10 (training: validation:test). I train on the training set then get 90% accuracy. I apply ...
3
votes
4answers
177 views

Metrics to determine K in K-cross fold validation

Consider a scenario where the dataset in hand is quite large, let's assume 50000 samples (quite well balanced between two classes). What metrics can be used to decide the K value in a K-fold cross-...
1
vote
1answer
30 views

Nested cross-validation generalization error for multiple models

I am referring to this question: Nested cross-validation and selecting the best regression model - is this the right SKLearn process? In the answers it shows that nested cv can estimate the ...
0
votes
2answers
75 views

Classification for High dimensional data

I am trying to build a classification model with 1000+ features and I am performing the below steps but I am not sure if it's the correct way to do it Step 1. Finding Variable Importance using the ...
3
votes
3answers
59 views

Folds in Cross validation

I am performing 10-folds cross-validation to evaluate the performances of a series of models (variable selection + regression) with R. I created manually the folds with this code. At the moment I'm ...
2
votes
2answers
56 views

Logloss vs Accuracy. Which needs to be chosen to evaluate the model performance

While model tuning using Cross validation and Grid search , I was plotting the graph of different learning rate against logloss and accuracy separately. Graph of Logloss --> learning Rate When I ...
0
votes
1answer
87 views

Target feature in training set or not?

If I analyse a random forest in python with scikit I do: ...
1
vote
0answers
31 views

Is this random forest logical correct and correct implemented with R and gbm?

For professional reasons I want to learn and understand random forests. I feel unsafe if my understanding is the correct or if I am doing logical errors. I got a data set with 15 million entries and ...