Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

1
vote
0answers
9 views

Optimizing decision threshold on model with oversampled/imbalanced data

I'm working on developing a model with a highly imbalanced dataset (0.7% Minority class). To remedy the imbalance, I was going to oversample using algorithms from imbalanced-learn library. I had a ...
2
votes
0answers
23 views

A classification machine learning flow chart implimenting dimentionality reduction, upsampling, and cross validation [closed]

I would like to make a flow chart for an ML classifier and make sure that my thinking is correct. Here is a little about my sample: I have 3 classes and about 160 features. I suspect that some of ...
2
votes
0answers
19 views

convolutional neural network with cross validation in Keras

I want to use K-fold cross-validation on my dataset of images. I am reading the data (images) from a directory. How do I use cross validation with convolutional neural network in Keras?
0
votes
0answers
12 views

Model validation in flexible parametric survival analysis: any special package or code in R or stata?

I am doing medical prediction research. I used to use logistic regression and Cox regression and I found my R or stata package to do model validation e.g. rms in R (validate function with bootstrap ...
1
vote
1answer
25 views

What is the best way to visualize 10 Fold Cross Validation Scores?

I have trained a CNN model and I have applied 10 Fold Cross Validation because I don't have much data to train the classifier. Now I am unsure about how to visulize fold wise results. Please suggest ...
1
vote
1answer
52 views

Using K-fold cross-validation in Keras on the data of my model

I would like to use K-fold cross-validation on my data of my model. My codes in Keras is : ...
1
vote
0answers
11 views

very large difference between cross_val and (multiple) r2 model evaluation

I did submit my first kaggle kernel, on the avocado dataset kernel link, I treated it like I should predict the avocado price so I splitted the dataset in a train & test set, fitted the model and ...
0
votes
0answers
26 views

Splitting images dataset using KFold

I have a small dataset of images and I want to use cross validation to train the images using deep learning model. I want to split the folder of images into different folders(folds). I want to split ...
0
votes
1answer
22 views

Confidence of this particual prediction

I am looking for a confidence of model to predict well in a given situation. So I have a model $f$ (generic, let's exemplify with a regression model of explicit form for brevity). It well fits on the ...
1
vote
1answer
36 views

Cross validation for anomaly detection using autoencoder

I am using autoencoder for anomaly detection in warranty data. I don't have any ground truth labels to confirm whether the anomalies detected by the model is really an anomaly or not. Since I don't ...
0
votes
1answer
32 views

How to do k-fold validation with classifiers?

I want to cross-validate a model that plays the card game below (see image). I trained the model on a dataset of 1000 games, with the goal to maximise the profit of each game. It works great on the ...
2
votes
1answer
19 views

K-fold crossvalidation: how do MSE average and variance vary with K?

I'd like to get an intuition about how varying k impacts k-fold validation. Is the following right? Average of the OOS MSEs should generally decrease with k Because, a bigger "k" means the training ...
0
votes
1answer
15 views

The Meaning Behind the Cross Validation Score in Factor Analysis

In order to choose the best number of underlying factors for my data using factor analysis, I decided to use the tutorial outlined in scikit-learn's documentation. Running ...
2
votes
0answers
109 views

K-fold cross validation when using fit_generator and flow_from_directory() in Keras

I am using flow_from_directory() and fit_generator in my deep learning model, and I want to use cross validation method to train ...
0
votes
0answers
12 views

How to perform platt scaling for hyperparameter-optimized model?

I'm using Python and have a best estimator from a grid search. Wanted to be able to calibrate the probability output accordingly, but would like to know more about implementing platt scaling. From ...
0
votes
1answer
21 views

sklearn cross_validate without test/train split

I'm running cross validation on my training data: ...
1
vote
1answer
26 views

Evaluating the performance of a random forest classifier

I'm using a random forest classifier (in R) to impute missing data in a dataset. Basically, I have a bunch of objects (companies) and I want to guess an attribute (...
2
votes
0answers
41 views

How to use SVD to reduce the dimension of test data which is not available at the time of SVD?

Usually, when both train and test data are available in the beginning, a dimensionality reduction such as Singular Value Decomposition (SVD) can be applied on both of them as one matrix. The reduced ...
0
votes
0answers
8 views

Strongly varying F1 score when training neural network multiple times

I have a simple feed forward network with 3 layers that I use to classify between 7 different classes. The input is a 100 or 300 dimensional (tried both) embedding vector that I already got from ...
1
vote
0answers
24 views

Nested cross-validation for regression over small dataset

I'm trying to do nested cross-validation for regression model parameter selection and prediction evaluation. I'm using temporal data (series of count). The problem is that I don't have a lot of data ...
0
votes
1answer
27 views

Some confusions on Model selection using cross-validation approach

https://stats.stackexchange.com/questions/11602/training-with-the-full-dataset-after-cross-validation explains the procedure and the importance of doing cross-validation to assess the performance of ...
1
vote
1answer
25 views

can accuracy rise while precision and recall drop?

I am working on a model and running some experiments, I see that under some configurations, The accuracy rises while the recall and precision are much lower, what is the mathematical explanation? is ...
0
votes
0answers
29 views

Model Selection with Oversampling/ Cross-Validation leads to similar test results in 2 approaches

Quick Intro Sorry for the long read. I added a lot in here because I wanted to describe what I've worked on so far, but I wanted to quickly summarize the issue I've been having, just so you have it ...
0
votes
1answer
45 views

Titanic Kaggle Data: Why am I getting lower accuracy on Kaggle submissions than on held-out data?

I am going through my first solo machine learning project and would like to gain some insight into what I am doing wrong/what is going on here as I am a bit stuck. I have been applying machine ...
1
vote
0answers
22 views

How to train and validate a model continously which affects its own future data?

We are working with a online marketplace. Our problem is to predict whether certain products are profitable or not for our marketplace in near future(next one month horizon). For example: Consider 2 ...
1
vote
1answer
42 views

Mean error (not squared) in scikit-learn cross_val_score [closed]

I need to know if the values generated by each fold of cross_val_score have a distribution which is centered on zero. Something as simple as the median or mean of <...
2
votes
1answer
98 views

cross_val_score meaning

I'm studying the following code, which cross_val_score_ was used as well as .mean() and .std(). I read many documentation of the meanings, but didn't get what each of the above does. ...
1
vote
0answers
13 views

EM-ELM Cross validation

I know that cross validation is used to find the best hyperparameters that minimize the average error. For example, the number of neurons that minimize the average error of cross-validation is ...
1
vote
2answers
91 views

Linear Regression + KFold cross validation

I have a prepossessed data set ready and the corresponding labels (8 classes). I've already done KFold cross validation with K=10 with some classifiers such as DT,KNN,NB and SVM and now I want to do ...
0
votes
0answers
83 views

Selecting a Specific Number of Features via Sklearn's RFECV (Recursive Feature Elimination with Cross-validation)

I'm wondering if it is possible for Sklearn's RFECV to select a fixed number of the most important features. For example, working on a dataset with 617 features, I have been trying to use RFECV to see ...
1
vote
2answers
79 views

Terminology - cross-validation, testing and validation set for classification task

Confusion1) If k=10 then does this mean that 90% is for training and 10% for testing? So always we have k% for testing? ...
0
votes
2answers
39 views

Is it possible to have a validation error less than train error for a while followed by the reverse behaviour?

I am solving for a regression (using tensorflow's DNNRegressor) problem. When I sampled out 20% data (randomly) and divided it further into train-eval (90-10%, random but mutually exclusive), I ...
1
vote
0answers
47 views

Expected behaviour of loss and accuracy when using data augmentation

I have implemented a convolutional neural network in Keras, and I use off-line data augmentation in the training set. The way I do this is that I create batches of training data in separate files (...
0
votes
2answers
115 views

Random Forest Classifier - KFold CV Tunes Very Deep Trees --> Overfitting?

I'm tuning a random forest in python and am wondering if/why my model is overfit. The dataset is described below: 1700 Positive Cases / 54000 total cases ~ 3.2% (unbalanced) 50 Numerical Features,~...
2
votes
1answer
27 views

Stratify on regression

I have worked in classification problems, and stratified cross-validation is one of the most useful and simple techniques I've found. In that case, what it means is to build a training and validation ...
0
votes
2answers
20 views

Validation set performance increased, test set performance decreased

I am training a CNN model for a three-class classification problem. To do this, I'm gradually unfreezing more convolutional blocks of a pre-trained Resnet-18 network. The thing is that after ...
0
votes
0answers
20 views

How to measure model improvement for Recommender Systems in real applications?

In the academia, model 'goodness' for recommendation systems are typically in terms of a loss or metric (i.e. MSE loss, Mean Average Precision). In real world applications, companies would deploy A/B ...
0
votes
1answer
68 views

Visualize the predicted and actual class after training and testing

The data set X has 10 features with 50 instances labelled as 0 and 1. Considering only 6 instances as an example here, let YPred ...
3
votes
1answer
36 views

Hyper parameters and ValidationSet

Please correct me if I am wrong. "Training Set is used for calculating parameters of a machine learning model, Validation data is used for calculating hyperparameters of the same model (we use same ...
0
votes
0answers
17 views

prediction and cross validation with rare factor levels

A problem I continually run into, in both prediction and cross validation (using R), is that some of the qualitative variables that I'm using for prediction via a model contain levels that were not ...
2
votes
1answer
30 views

How to select the learned model using $k$-fold cross validation?

Let us consider a case where $1000$ data is given, i.e., the data set $U=\{x_1, \ldots, x_{1000}\}.$ When we want to use $k$-fold validation scheme, we first divide the data set into $k$ groups. ...
2
votes
1answer
114 views

GridSearchCV results are different to directly applied default model (SVM)

I run a Support Vector Machines model on part of my train set with following result: ...
2
votes
3answers
342 views

What is the difference between bootstrapping and cross-validation?

I used to apply K-fold cross-validation for robust evaluation of my machine learning models. But I'm aware of the existence of the bootstrapping method for this purpose as well. However, I cannot see ...
3
votes
1answer
25 views

Validation of a model generated by automated deep learning system

How would one go about selecting the validation set used to evaluate trained models by an automated system, in order to ensure that each new model is at least as good as, or better than the previous ...
0
votes
1answer
47 views

Cross Validation and training set

In RapidMiner I want to create a k-NN model in order to create a classifier. To generate the test sets and the training sets I use the the cross-validation. If I choose 10 as the number of folds, the ...
-1
votes
1answer
31 views

CV hyperparameter in sklearn.model_selection.cross_validate

I've got a problem with understanding the CV parameter in cross_validate. Could you check if I understand it correctly? I'm running ML algorithms in big set of data (train 37M rows), therefore I ...
1
vote
2answers
58 views

Why can't I choose my hyper-parameter in the training set?

Say I've divided the data into 3 parts: training, validation and test. I know for example, that in Neural Networks, the number of hidden layers is a hyper parameter. Why can't I train numerous NN ...
1
vote
1answer
29 views

Fitting and transforming text data in training, testing, and validation sets

I'm trying to implement a simple text classifier wherein the data is split into training (70%) and testing (30%) sets, but cross validation (k=10) to be performed on the training set. My main concern ...
0
votes
0answers
7 views

How can I forward propagate my whole training set many times before updating the weights in keras?

I am training a network in keras with tensorflow backend. Using scikit-learn for Hyperparameter optimization using crossvalidation. I have a very limited sample size. The idea is to increase the ...
2
votes
1answer
49 views

Question about K-Fold Cross Validation

In a machine learning procedure, suppose we've chosen k=10 for the "K-Fold Cross Validation". After we do the k steps of "K-Fold Cross Validation", how do we choose the final model for the classifier ?...