Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

Does sklearn.pipeline have a single mechanism for cross-validation regardless of model API?

With a single standard interface (sklearn.pipeline) on top of different regressors, how do I use cross-validation? The example below uses two regressors with different internal cross-validation ...
0
votes
3answers
43 views

Need for cross-validation in KNN

I read that we need cross-validation in KNN algorithm as the K value that we have found from the TRAIN-TEST of KNN might not be generalizable on unseen data. The logic given was that, the TEST data ...
0
votes
0answers
14 views

Find train sample with features distribution similar to test

Assume that I have two datasets $Train$ and $Test$. And there is the problem illustrated below: there are different feature distribution between two datasets I want to find the train subset $A \...
0
votes
1answer
27 views

Custom vectorizer transformer in sklearn with cross validation

I created a custom transformer class called Vectorizer() that inherits from sklearn's ...
1
vote
1answer
38 views

The behavior of the cross validation error and training error in underfitting case is not clear

I currently study the "Machine Learning" course on Coursera.org by Andrew Ng, it comes to a topic that discusses the performance of learning algorithms under different conditions. Here, we ...
1
vote
1answer
14 views

Random Forest Model Train, Save and Predict Later vs Train and Predict Right Away - Different Results

I tested two pieces of code and they delivered different results, which was quite unexpected. First piece of code is supposed to train models in a k-fold manner, preserve each one of these fitted ...
0
votes
0answers
26 views

Split dataframe to a train and test sets with a cross validation of x%

I am working on a dataframe and need to split it into a training set and test set, with 90% for Cross-Validation training, and 10% for a final test set. The problem is that I do not know where to ...
0
votes
0answers
43 views

How to evaluate model accuracy at tail of empirical distribution?

I am making a nonlinear regression on stationary dependent variable and I want to precisely forecast extreme values of this variable. So when my model predicts extreme values I want them to be highly ...
0
votes
2answers
47 views

validation/test set uniqueness question

Hopefully a simple question, but it's a little unclear to me on how best to separate train/validate/test sets. I have say 100 examples of class A. I'm classifying text into either class A, which I ...
1
vote
0answers
20 views

Interpreting XGBoost's results (when they span between [0,0.5])

I would like to classify sentences into one of two different categories. I trained a XGBoost model over a search grid with k-folds cross-validation. My data represents sentences, and features ...
0
votes
0answers
10 views

Cross-validation with time series data and Deep Learning Models

I have in a pandas dataframe two columns: "GDP" and "Unemployment Rate", from 1950 until now. I'm trying to apply a simple RNN model to this data, where we would use past and ...
0
votes
0answers
6 views

Implement NestedCV into Neural Networks

I have a regression task for which I am using ML models. My input features are 64. I implement NestedCV to get best ML models and hyperparameters. I have recently learned Neural Networks and want to ...
2
votes
1answer
34 views

Using cross validation score to perform feature selection

So to perform my feature selection I ran cross validation over and over again, each time trying different subsets of my attributes and repeated this until I got the best cross validation score I could ...
0
votes
0answers
17 views

For a binary classification algorithm, is there an objective way to know how large your set of positive and negative labels need to be?

We're training a binary classification algorithm using a combined total of 2000 positive and negative labels that we purchased from a data vendor. We mostly used all the textbook machine learning ...
1
vote
1answer
38 views

In multi-class, is the average accuracy of each class in confusion metrics equal to the accuracy calculated from cross validation?

When I calculate accuracy through cross-validation, it gives me a different accuracy than when I calculate through confusion metrics. Why does it give different accuracy? Is overall calculated ...
0
votes
0answers
14 views

Removing attributes prior to nested cross validation feature selection

I am using nested cross validation for feature selection and hyperparameter tuning. I find that the inner loop is quite consistent and often picks the attributes A,B,C,D,E (renamed for this question) ...
0
votes
1answer
35 views

K-Fold cross validating with random forest - how to correctly fit model to every fold?

So I have created K-Folds from my data using this code: ...
2
votes
1answer
43 views

Cross validation and hyperparameter tuning workflow

After reading a lot of articles on cross validation, I am now confused. I know that cross validation is used to get an estimate of model performance and is used to select the best algorithm out of ...
0
votes
0answers
12 views

Are my classification results like accuracy, precision, recall etc significant and valid for general data?

So I have this data let's say of size (2000,11), and I want to do perform a binary classification based on these eleven features. There is a class Imbalance between ...
1
vote
1answer
28 views

Difference between model score on test part and Kaggle public score

I tested my CatBoostModel model on part of data and get 0.92 score, but Kaggle public score was 0.9. I found new hyperparameters via randomsearch, new model score was 0.925, but on Kaggle score fell ...
1
vote
1answer
17 views

Is there any better approach then K folds and nested K folds?

I am trying to understand what problem is K-folds solving. It does not seem to be solving data leakage at all, as we are still testing on test data and then taking an average of all test folds and ...
1
vote
1answer
44 views

Python scikit learn KFold function uneven train, test split

i have the following code below where i have noticed that the length of the train, test split from Kfold.split() is different for the last fold. Any reason why this may be happening and how i can go ...
1
vote
1answer
41 views

Poor binary text classification results

I have a binary NLP classification task to identify text that talks about a target topic from millions of sentences. Between 5-10% of sentences are positive, the rest is negative. I have trained ...
1
vote
0answers
42 views

Cross Validation in Neural Networks

I am training a neural network and doing 10-fold cross validation to measure performance. I have read lots of documentation and forums telling that the set of weights that should be saved or ...
0
votes
1answer
47 views

Is it necessary to use stratified sampling if I am using SMOTE already?

I have already applied SMOTE to my imbalanced dataset with more than 300K observations. Does it still make sense to use stratified K-fold cross validation rather than simply ordinary K-fold cross ...
1
vote
0answers
18 views

How should you preprocess the data before K-fold validation?

I often see Kaggle notebook authors preprocessing the entire training data prior to splitting it for K-fold validation, but does this have a risk of leaking information into the validation set each ...
1
vote
0answers
27 views

How to implement kfold and cv into Hybrid feature selection and evaluate the classification model performance?

I have been working on a Hybrid feature selection combined with hyperopt package for hyperparameter tuning and I am thinking about evaluating the performance of several model classifiers. I looked ...
1
vote
2answers
38 views

Understanding Sklearns learning_curve

I have been using sklearns learning_curve , and there are a few questions I have that are not answered by the documentation(see also here and here), as well as questions that are raised by the ...
0
votes
1answer
36 views

Query regarding surprising spike in accuracy of ML model

I implemented all the major ML models (Logistic Regression, Naive Bayes, SVM, KNN, Decision Tree, Random Forest, Ada Boost & XGBoost) on my dataset. My stratified cross-validation scores are ...
0
votes
1answer
27 views

Comparing accuracies of Grid Search CV & Randomized Search CV with K-Fold Cross Validation?

Are Grid Search CV & Randomized Search CV always/necessarily supposed to give more accurate results after hyperparameter tuning as compared to K-Fold Cross Validation?
0
votes
1answer
33 views

Steps of multiclass classification problem

So this question is more theoretical, than a practical one. I got a dataframe with 4 classes of cars' body types (e.g. sedan, hatchback, etc.) and different characteristics (doors, seats, maximum ...
2
votes
2answers
118 views

Removing the outliers improved my models, is what I did good or bad?

I used cross validation on my data (11000 rows) with maximum salary of 10000 and after some cleaning I got to rmse=70. Then I tried to remove the outliers 10 times just to try things now I have 9000 ...
1
vote
1answer
59 views

Splitting the dataset manually for k-Fold Cross-Validation

I manually divided the dataset into three sets: train, test, and validation. Each set includes several folders, one for each patient. Each patient has many images from a different point of view. As a ...
0
votes
2answers
67 views

I am attempting to implement k-folds cross validation in python3. What is the best way to implement this? Is it preferable to use Pandas or Numpy? [closed]

I am attempting to create a script to implement cross validation in data. However, the splits cannot randomly take any records, so the training and testing can be done on equal data splits for each ...
1
vote
2answers
46 views

How to get best data split from cross validation

I have trained a Random forest regressor which is giving me a rmse score of 70.72. But when I tried the same model in cross_val_score with a cv of 10, it gave me an array that looks something like ...
1
vote
1answer
59 views

Cross-validation split for modelling data with timeseries behavior

Background: I have a dataset that is generated every month (it is similar with card data that contains card demography and transactions every month and new accounts can be added in the middle of data ...
2
votes
1answer
27 views

leave one pair out cross validation

I am trying to train and validate my datasets which contains 17 datasets. I have divided them as 15 for training and 2 for validation. In the process, I train on 15 datasets and use the generated ...
1
vote
1answer
29 views

How can I use the K-fold cross validation for one-class classifier? [closed]

I train a one-class classifier by Class X. In the testing stage, I use Class X and Class Y for validation. I want to compute the F-score metric of Class Y. How can I use the K-fold cross-validation ...
1
vote
2answers
45 views

Why cant we further tune/change the model after evaluating on the test set?

Every thread on stackexchange that I've found says that you can only use the test set once and thats it. So for instance, if you used a linear regression model and got poor results on the test set, ...
0
votes
0answers
49 views

Building a custom scoring function to find mean time-dependent AUC

I’m working on a survival analysis to predict 1-year mortality. I’m trying to build a custom score function that maximizes mean time-dependent AUC. Here is a description of the time-dependent AUC ...
0
votes
0answers
18 views

Qaulity assurance framework for ML/AI

I'd like to hear how organisations are applying a quality framework to ML/AI work. I'm struggling to find good content or good practice on this. By saying quality frameworks, for example, if I was ...
0
votes
1answer
183 views

New to keras neural network and k-fold cross validation

I'm new to learning neural networks and I found an example online to test accuracy with k-fold cross validation. The example is for binary data but I want to test MAE or RMSE (I guess?) for my ...
0
votes
0answers
18 views

Training and validation accuracy stagnating after a few epochs for text embeddings

I have text embeddings (768 dimensional vectors). I tried to build a feed forward neural network on classify the text into two classes. The network I used. ...
0
votes
0answers
36 views

Cross-validation for Random forest

I want to use cross-validation method for Random forest and compare the performance with other algorithms in the same condition. I heard from somebody that Random forest does not need cross-validation....
0
votes
1answer
81 views

Learning curves

I am working on a multiclass classification problem. I want to know whether my model is overfitting or underfitting. I am learning how to plot learning curves and have 4 doubts. 1.) Is the ordering of ...
0
votes
0answers
22 views

LSTM Data Preparation Input Shape

I have a 2-D dataframe df: ...
1
vote
1answer
50 views

Perfect scores for multiclass classification

I am working on a multiclass classification problem with 3 (1, 2, 3) classes being perfectly distributed. (70 instances of each class resulting in (210, 8) dataframe). Now my data has all the 3 ...
0
votes
0answers
24 views

Time-Series Cross-Validation for LSTM

Is it at all possible to separate my data into train/test sets with cross validation for time series data? I am experimenting with a LSTM model. Also, I am hoping to prevent data leakage/peaking in ...
3
votes
3answers
393 views

Model performance worsens after Cross Validation

I am training a logistic regression model on a dataset with only numerical features. I performed the following steps:- 1.) heatmap to remove collinearity between variables 2.) scaling using ...
0
votes
0answers
12 views

Model on non-iid data performing badly

I am on this lecture about non-iid data where we generated a timeseries data using the function below: ...

1
2 3 4 5
12