Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

0
votes
0answers
5 views

Does sklearn's gridsearchCV use the same cross validation train/test splits for evaluating each hyperparameter combination?

I couldn't find the answer on any forum in the interwebs so I hunted down the answer myself. Yes; the same train/test splits are used for each parameter combination. Here Is the relevant sklearn ...
0
votes
0answers
9 views

Name for non-exhaustive cross validation technique similar to leave-one-out but with test set size larger than one

Is there a name for a cross validation technique like k-fold that accepts the size of the fold instead of the amount of folds? If I understand it correctly, leave-p-out outputs all combinations, so ...
0
votes
1answer
12 views

Dataset for studying cross-validation techniques [closed]

can you recommend a dataset that can be used to study/ illustrate cross-validation techniques such as parameter tuning, nested-cross validation, repeated cross validation etc. For instance, is there ...
1
vote
0answers
9 views

Strangeness in validation loss between CPU vs GPU when training CNN

I've been training an implementation of Mask R-CNN and it was training very successfully on my CPU but I've just set up my GPU and it is giving some strange results when looking at my validation loss. ...
0
votes
0answers
28 views

Are my Random Forest Classifier and Regressors overfitting?? I have CV and learning curves!

I seem to be getting great results from logistic regression with RFE and random forest feature importances in support, but there's been a suggestion of overfitting and when I run learning curves the ...
1
vote
1answer
23 views

Please help select an Algo based on Accuracy and Confusion Matrix

I am very new to Data Science would appreciate your advice big time. Got a task: predict if a trade will be profitable or not, based on a set of data. I have prepared, cleaned and tested data. ...
0
votes
1answer
15 views

Can accuracy become worse on the training set with more epochs?

I know that overfitting occurs when the accuracy on the training set improves but the accuracy on the validation set decrease. So, we must stop the training. I would like to know if this is a rule ...
0
votes
2answers
44 views

Validation accuracy is always close to training accuracy

I am trying to tune the hyperparameters of a LSTM I have to do time series forecasting. I have noticed that my validation accuracy is always very close to my training accuracy. I am not sure whether ...
1
vote
0answers
18 views

How do I perform Leave One Out Cross Validation For Top n Recommendation Sytems?

I am new in making recommendation systems . I am using the surpriselib library to evaluate my recommendations. All the Accuracy Metrics are well supported in this library. But I also want to compute ...
1
vote
1answer
58 views

cross validation for small dataset

I have a dataset of 39 medical MR images, and I have to build a model to classify the tumor type. so is it suitable to use k-fold cross validation for validating the model? if so, what would be the ...
1
vote
1answer
82 views

A few questions to understand a random forest blog [closed]

I'm trying to understand a nice blog on the trade-off between sensitivity versus specificity with the random forest and logistic regression models. I have a few questions: 1) The blog used a 10 fold ...
5
votes
4answers
136 views

Which is first ? Tuning the parameters or selecting the model

I've been reading about how we split our data into 3 parts; generally, we use the validation set to help us tune the parameters and the test set to have an unbiased estimate on how well does our model ...
-1
votes
0answers
75 views

Random Forest with cross validation using TreeBagger on Matlab

I'd like to do cross validation on a Random Forest model. I've tried using crossval but it doesn't work on TreeBagger. I tried using for loop, but I'm not sure it's correct: ...
1
vote
0answers
38 views

100% classification accuracy

I am trying to perform a multi-class classification where the network is trained to classify objects into 3 categories: cars, pedestrians and miscellaneous. I am using the KITTI Dataset for car ...
1
vote
1answer
43 views

How to apply Stacking cross validation for time-series data?

Normally stacking algorithm uses K-fold cross validation technique to predict oof validation that used for level 2 prediction. In case of time-series data (say stock movement prediction), K-fold ...
3
votes
1answer
66 views

Hyperparameter tuning for stacked models

I'm reading the following kaggle post for learning how to incorporate model stacking http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/ in ML models. The structure ...
1
vote
1answer
31 views

Is the ultimate challenge in ML simply computational power?

I am stuck on a theoretical roadblock in learning about machine learning, because I have not seen this explicitly addressed anywhere. In my studies, it seems as if Cross-validation (or some variant ...
0
votes
1answer
42 views

How can one use a validation set to reduce overfitting Naive Bayes?

What is the correct procedure for using a validation set to reduce overfitting? Say I split the data 80:10:10 (training: validation:test). I train on the training set then get 90% accuracy. I apply ...
3
votes
4answers
154 views

Metrics to determine K in K-cross fold validation

Consider a scenario where the dataset in hand is quite large, let's assume 50000 samples (quite well balanced between two classes). What metrics can be used to decide the K value in a K-fold cross-...
1
vote
1answer
19 views

Nested cross-validation generalization error for multiple models

I am referring to this question: Nested cross-validation and selecting the best regression model - is this the right SKLearn process? In the answers it shows that nested cv can estimate the ...
0
votes
2answers
68 views

Classification for High dimensional data

I am trying to build a classification model with 1000+ features and I am performing the below steps but I am not sure if it's the correct way to do it Step 1. Finding Variable Importance using the ...
3
votes
3answers
57 views

Folds in Cross validation

I am performing 10-folds cross-validation to evaluate the performances of a series of models (variable selection + regression) with R. I created manually the folds with this code. At the moment I'm ...
0
votes
0answers
72 views

How does `cvpartition` in Matlab work?

I have split my entire dataset X and the label set Y into Xtrain_val and ...
0
votes
0answers
16 views

How to choose Elastic-Net parameters for feature selection?

I recently came across using elastic nets for feature selection which brings in regularization to temper the sparsity properties of L1 regressions. I would like to learn how to use elastic nets for ...
2
votes
2answers
50 views

Logloss vs Accuracy. Which needs to be chosen to evaluate the model performance

While model tuning using Cross validation and Grid search , I was plotting the graph of different learning rate against logloss and accuracy separately. Graph of Logloss --> learning Rate When I ...
0
votes
1answer
46 views

Target feature in training set or not?

If I analyse a random forest in python with scikit I do: ...
1
vote
0answers
30 views

Is this random forest logical correct and correct implemented with R and gbm?

For professional reasons I want to learn and understand random forests. I feel unsafe if my understanding is the correct or if I am doing logical errors. I got a data set with 15 million entries and ...
2
votes
0answers
20 views

Is it possible to get the mean coefficent of regression after “Test & Score” in Orange using cross-validation?

The cross-validation devides the data into n folds and measures the accuracy n times therefore. The displayed RMSE in the Test & Score tool is a mean of all n runs, I think. But is it possible to ...
1
vote
0answers
7 views

Validating performance of panel data based models

I'm wondering from a theoretical/general practice perspective, what is the best way to evaluate performance of regression models derived from panel data (i.e. a time series of cross sectional data). ...
0
votes
1answer
847 views

Validation-split of Keras fit function

Validation-split in Keras Sequential model fit function is documented as following on https://keras.io/models/sequential/ : validation_split: Float between 0 and 1. Fraction of the training data ...
0
votes
1answer
27 views

What do I initialise each model in cross validation with in a multi-layer Perceptron?

So, as far as my understanding goes, cross-validation is used to determine the best model. I understand that once we determine the best model, we then train it on the entire dataset. I'm supposed to ...
1
vote
0answers
27 views

Decent ROC, but horrible Precision-Recall curve

I was working on a model with following process: Split to training/validation/test sets Try a series of different models like GBM, RF, Logistic Regressions Optimize hyper-params on them using ...
1
vote
1answer
34 views

Optimizing decision threshold on model with oversampled/imbalanced data

I'm working on developing a model with a highly imbalanced dataset (0.7% Minority class). To remedy the imbalance, I was going to oversample using algorithms from imbalanced-learn library. I had a ...
2
votes
0answers
42 views

A classification machine learning flow chart implimenting dimentionality reduction, upsampling, and cross validation [closed]

I would like to make a flow chart for an ML classifier and make sure that my thinking is correct. Here is a little about my sample: I have 3 classes and about 160 features. I suspect that some of ...
3
votes
0answers
143 views

convolutional neural network with cross validation in Keras

I want to use K-fold cross-validation on my dataset of images. I am reading the data (images) from a directory. How do I use cross validation with convolutional neural network in Keras?
1
vote
1answer
191 views

What is the best way to visualize 10 Fold Cross Validation Scores?

I have trained a CNN model and I have applied 10 Fold Cross Validation because I don't have much data to train the classifier. Now I am unsure about how to visulize fold wise results. Please suggest ...
1
vote
1answer
973 views

Using K-fold cross-validation in Keras on the data of my model

I would like to use K-fold cross-validation on my data of my model. My codes in Keras is : ...
1
vote
0answers
27 views

very large difference between cross_val and (multiple) r2 model evaluation

I did submit my first kaggle kernel, on the avocado dataset kernel link, I treated it like I should predict the avocado price so I splitted the dataset in a train & test set, fitted the model and ...
0
votes
0answers
82 views

Splitting images dataset using KFold

I have a small dataset of images and I want to use cross validation to train the images using deep learning model. I want to split the folder of images into different folders(folds). I want to split ...
0
votes
1answer
30 views

Confidence of this particual prediction

I am looking for a confidence of model to predict well in a given situation. So I have a model $f$ (generic, let's exemplify with a regression model of explicit form for brevity). It well fits on the ...
1
vote
2answers
139 views

Cross validation for anomaly detection using autoencoder

I am using autoencoder for anomaly detection in warranty data. I don't have any ground truth labels to confirm whether the anomalies detected by the model is really an anomaly or not. Since I don't ...
0
votes
1answer
46 views

How to do k-fold validation with classifiers?

I want to cross-validate a model that plays the card game below (see image). I trained the model on a dataset of 1000 games, with the goal to maximise the profit of each game. It works great on the ...
3
votes
1answer
46 views

K-fold crossvalidation: how do MSE average and variance vary with K?

I'd like to get an intuition about how varying k impacts k-fold validation. Is the following right? Average of the OOS MSEs should generally decrease with k Because, a bigger "k" means the training ...
0
votes
1answer
39 views

The Meaning Behind the Cross Validation Score in Factor Analysis

In order to choose the best number of underlying factors for my data using factor analysis, I decided to use the tutorial outlined in scikit-learn's documentation. Running ...
3
votes
0answers
585 views

K-fold cross validation when using fit_generator and flow_from_directory() in Keras

I am using flow_from_directory() and fit_generator in my deep learning model, and I want to use cross validation method to train ...
1
vote
0answers
53 views

How to perform platt scaling for hyperparameter-optimized model?

I'm using Python and have a best estimator from a grid search. Wanted to be able to calibrate the probability output accordingly, but would like to know more about implementing platt scaling. From ...
0
votes
1answer
219 views

sklearn cross_validate without test/train split

I'm running cross validation on my training data: ...
1
vote
1answer
38 views

Evaluating the performance of a random forest classifier

I'm using a random forest classifier (in R) to impute missing data in a dataset. Basically, I have a bunch of objects (companies) and I want to guess an attribute (...
2
votes
0answers
132 views

How to use SVD to reduce the dimension of test data which is not available at the time of SVD?

Usually, when both train and test data are available in the beginning, a dimensionality reduction such as Singular Value Decomposition (SVD) can be applied on both of them as one matrix. The reduced ...