Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

1
vote
0answers
7 views

Hyperparameter tuning for stacked models

I'm reading the following kaggle post for learning how to incorporate model stacking http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/ in ML models. The structure ...
0
votes
1answer
20 views

Is the ultimate challenge in ML simply computational power?

I am stuck on a theoretical roadblock in learning about machine learning, because I have not seen this explicitly addressed anywhere. In my studies, it seems as if Cross-validation (or some variant ...
0
votes
1answer
16 views

How can one use a validation set to reduce overfitting Naive Bayes?

What is the correct procedure for using a validation set to reduce overfitting? Say I split the data 80:10:10 (training: validation:test). I train on the training set then get 90% accuracy. I apply ...
3
votes
4answers
126 views

Metrics to determine K in K-cross fold validation

Consider a scenario where the dataset in hand is quite large, let's assume 50000 samples (quite well balanced between two classes). What metrics can be used to decide the K value in a K-fold cross-...
1
vote
1answer
10 views

Nested cross-validation generalization error for multiple models

I am referring to this question: Nested cross-validation and selecting the best regression model - is this the right SKLearn process? In the answers it shows that nested cv can estimate the ...
0
votes
1answer
38 views

Classification for High dimensional data

I am trying to build a classification model with 1000+ features and I am performing the below steps but I am not sure if it's the correct way to do it Step 1. Finding Variable Importance using the ...
3
votes
3answers
53 views

Folds in Cross validation

I am performing 10-folds cross-validation to evaluate the performances of a series of models (variable selection + regression) with R. I created manually the folds with this code. At the moment I'm ...
0
votes
0answers
23 views

How does `cvpartition` in Matlab work?

I have split my entire dataset X and the label set Y into Xtrain_val and ...
0
votes
0answers
13 views

How to choose Elastic-Net parameters for feature selection?

I recently came across using elastic nets for feature selection which brings in regularization to temper the sparsity properties of L1 regressions. I would like to learn how to use elastic nets for ...
1
vote
1answer
29 views

Logloss vs Accuracy. Which needs to be chosen to evaluate the model performance

While model tuning using Cross validation and Grid search , I was plotting the graph of different learning rate against logloss and accuracy separately. Graph of Logloss --> learning Rate When I ...
-1
votes
0answers
17 views

Feature selection, hyperparameters tuning and model test with Cross Validation

I don't know if this is a valid question, but I am not finding any documentation or explanation anywhere and I have a doubt. I have a dataset - 100 samples, 50 variables, 3 response variables (1 ...
0
votes
1answer
26 views

Target feature in training set or not?

If I analyse a random forest in python with scikit I do: ...
1
vote
0answers
27 views

Is this random forest logical correct and correct implemented with R and gbm?

For professional reasons I want to learn and understand random forests. I feel unsafe if my understanding is the correct or if I am doing logical errors. I got a data set with 15 million entries and ...
2
votes
0answers
12 views

Is it possible to get the mean coefficent of regression after “Test & Score” in Orange using cross-validation?

The cross-validation devides the data into n folds and measures the accuracy n times therefore. The displayed RMSE in the Test & Score tool is a mean of all n runs, I think. But is it possible to ...
1
vote
0answers
6 views

Validating performance of panel data based models

I'm wondering from a theoretical/general practice perspective, what is the best way to evaluate performance of regression models derived from panel data (i.e. a time series of cross sectional data). ...
0
votes
1answer
90 views

Validation-split of Keras fit function

Validation-split in Keras Sequential model fit function is documented as following on https://keras.io/models/sequential/ : validation_split: Float between 0 and 1. Fraction of the training data ...
0
votes
1answer
22 views

What do I initialise each model in cross validation with in a multi-layer Perceptron?

So, as far as my understanding goes, cross-validation is used to determine the best model. I understand that once we determine the best model, we then train it on the entire dataset. I'm supposed to ...
1
vote
0answers
24 views

Decent ROC, but horrible Precision-Recall curve

I was working on a model with following process: Split to training/validation/test sets Try a series of different models like GBM, RF, Logistic Regressions Optimize hyper-params on them using ...
1
vote
1answer
23 views

Optimizing decision threshold on model with oversampled/imbalanced data

I'm working on developing a model with a highly imbalanced dataset (0.7% Minority class). To remedy the imbalance, I was going to oversample using algorithms from imbalanced-learn library. I had a ...
2
votes
0answers
29 views

A classification machine learning flow chart implimenting dimentionality reduction, upsampling, and cross validation [closed]

I would like to make a flow chart for an ML classifier and make sure that my thinking is correct. Here is a little about my sample: I have 3 classes and about 160 features. I suspect that some of ...
3
votes
0answers
85 views

convolutional neural network with cross validation in Keras

I want to use K-fold cross-validation on my dataset of images. I am reading the data (images) from a directory. How do I use cross validation with convolutional neural network in Keras?
0
votes
0answers
18 views

Model validation in flexible parametric survival analysis: any special package or code in R or stata?

I am doing medical prediction research. I used to use logistic regression and Cox regression and I found my R or stata package to do model validation e.g. rms in R (validate function with bootstrap ...
1
vote
1answer
71 views

What is the best way to visualize 10 Fold Cross Validation Scores?

I have trained a CNN model and I have applied 10 Fold Cross Validation because I don't have much data to train the classifier. Now I am unsure about how to visulize fold wise results. Please suggest ...
1
vote
1answer
333 views

Using K-fold cross-validation in Keras on the data of my model

I would like to use K-fold cross-validation on my data of my model. My codes in Keras is : ...
1
vote
0answers
22 views

very large difference between cross_val and (multiple) r2 model evaluation

I did submit my first kaggle kernel, on the avocado dataset kernel link, I treated it like I should predict the avocado price so I splitted the dataset in a train & test set, fitted the model and ...
0
votes
0answers
54 views

Splitting images dataset using KFold

I have a small dataset of images and I want to use cross validation to train the images using deep learning model. I want to split the folder of images into different folders(folds). I want to split ...
0
votes
1answer
28 views

Confidence of this particual prediction

I am looking for a confidence of model to predict well in a given situation. So I have a model $f$ (generic, let's exemplify with a regression model of explicit form for brevity). It well fits on the ...
1
vote
2answers
89 views

Cross validation for anomaly detection using autoencoder

I am using autoencoder for anomaly detection in warranty data. I don't have any ground truth labels to confirm whether the anomalies detected by the model is really an anomaly or not. Since I don't ...
0
votes
1answer
37 views

How to do k-fold validation with classifiers?

I want to cross-validate a model that plays the card game below (see image). I trained the model on a dataset of 1000 games, with the goal to maximise the profit of each game. It works great on the ...
2
votes
1answer
29 views

K-fold crossvalidation: how do MSE average and variance vary with K?

I'd like to get an intuition about how varying k impacts k-fold validation. Is the following right? Average of the OOS MSEs should generally decrease with k Because, a bigger "k" means the training ...
0
votes
1answer
17 views

The Meaning Behind the Cross Validation Score in Factor Analysis

In order to choose the best number of underlying factors for my data using factor analysis, I decided to use the tutorial outlined in scikit-learn's documentation. Running ...
3
votes
0answers
323 views

K-fold cross validation when using fit_generator and flow_from_directory() in Keras

I am using flow_from_directory() and fit_generator in my deep learning model, and I want to use cross validation method to train ...
0
votes
0answers
21 views

How to perform platt scaling for hyperparameter-optimized model?

I'm using Python and have a best estimator from a grid search. Wanted to be able to calibrate the probability output accordingly, but would like to know more about implementing platt scaling. From ...
0
votes
1answer
126 views

sklearn cross_validate without test/train split

I'm running cross validation on my training data: ...
1
vote
1answer
34 views

Evaluating the performance of a random forest classifier

I'm using a random forest classifier (in R) to impute missing data in a dataset. Basically, I have a bunch of objects (companies) and I want to guess an attribute (...
2
votes
0answers
82 views

How to use SVD to reduce the dimension of test data which is not available at the time of SVD?

Usually, when both train and test data are available in the beginning, a dimensionality reduction such as Singular Value Decomposition (SVD) can be applied on both of them as one matrix. The reduced ...
0
votes
0answers
8 views

Strongly varying F1 score when training neural network multiple times

I have a simple feed forward network with 3 layers that I use to classify between 7 different classes. The input is a 100 or 300 dimensional (tried both) embedding vector that I already got from ...
1
vote
0answers
35 views

Nested cross-validation for regression over small dataset

I'm trying to do nested cross-validation for regression model parameter selection and prediction evaluation. I'm using temporal data (series of count). The problem is that I don't have a lot of data ...
0
votes
1answer
33 views

Some confusions on Model selection using cross-validation approach

https://stats.stackexchange.com/questions/11602/training-with-the-full-dataset-after-cross-validation explains the procedure and the importance of doing cross-validation to assess the performance of ...
1
vote
1answer
30 views

can accuracy rise while precision and recall drop?

I am working on a model and running some experiments, I see that under some configurations, The accuracy rises while the recall and precision are much lower, what is the mathematical explanation? is ...
0
votes
0answers
58 views

Model Selection with Oversampling/ Cross-Validation leads to similar test results in 2 approaches

Quick Intro Sorry for the long read. I added a lot in here because I wanted to describe what I've worked on so far, but I wanted to quickly summarize the issue I've been having, just so you have it ...
0
votes
1answer
75 views

Titanic Kaggle Data: Why am I getting lower accuracy on Kaggle submissions than on held-out data?

I am going through my first solo machine learning project and would like to gain some insight into what I am doing wrong/what is going on here as I am a bit stuck. I have been applying machine ...
1
vote
0answers
24 views

How to train and validate a model continously which affects its own future data?

We are working with a online marketplace. Our problem is to predict whether certain products are profitable or not for our marketplace in near future(next one month horizon). For example: Consider 2 ...
1
vote
1answer
81 views

Mean error (not squared) in scikit-learn cross_val_score [closed]

I need to know if the values generated by each fold of cross_val_score have a distribution which is centered on zero. Something as simple as the median or mean of <...
2
votes
1answer
309 views

cross_val_score meaning

I'm studying the following code, which cross_val_score_ was used as well as .mean() and .std(). I read many documentation of the meanings, but didn't get what each of the above does. ...
1
vote
0answers
17 views

EM-ELM Cross validation

I know that cross validation is used to find the best hyperparameters that minimize the average error. For example, the number of neurons that minimize the average error of cross-validation is ...
2
votes
2answers
482 views

Linear Regression + KFold cross validation

I have a prepossessed data set ready and the corresponding labels (8 classes). I've already done KFold cross validation with K=10 with some classifiers such as DT,KNN,NB and SVM and now I want to do ...
0
votes
0answers
174 views

Selecting a Specific Number of Features via Sklearn's RFECV (Recursive Feature Elimination with Cross-validation)

I'm wondering if it is possible for Sklearn's RFECV to select a fixed number of the most important features. For example, working on a dataset with 617 features, I have been trying to use RFECV to see ...
1
vote
2answers
111 views

Terminology - cross-validation, testing and validation set for classification task

Confusion1) If k=10 then does this mean that 90% is for training and 10% for testing? So always we have k% for testing? ...
0
votes
2answers
60 views

Is it possible to have a validation error less than train error for a while followed by the reverse behaviour?

I am solving for a regression (using tensorflow's DNNRegressor) problem. When I sampled out 20% data (randomly) and divided it further into train-eval (90-10%, random but mutually exclusive), I ...