Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

Implement NestedCV into Neural Networks

I have a regression task for which I am using ML models. My input features are 64. I implement NestedCV to get best ML models and hyperparameters. I have recently learned Neural Networks and want to ...
2
votes
1answer
19 views

Using cross validation score to perform feature selection

So to perform my feature selection I ran cross validation over and over again, each time trying different subsets of my attributes and repeated this until I got the best cross validation score I could ...
0
votes
0answers
16 views

For a binary classification algorithm, is there an objective way to know how large your set of positive and negative labels need to be?

We're training a binary classification algorithm using a combined total of 2000 positive and negative labels that we purchased from a data vendor. We mostly used all the textbook machine learning ...
1
vote
0answers
22 views

In multi-class, is the average accuracy of each class in confusion metrics equal to the accuracy calculated from cross validation?

When I calculate accuracy through cross-validation, it gives me a different accuracy than when I calculate through confusion metrics. Why does it give different accuracy? Is overall calculated ...
0
votes
0answers
14 views

Removing attributes prior to nested cross validation feature selection

I am using nested cross validation for feature selection and hyperparameter tuning. I find that the inner loop is quite consistent and often picks the attributes A,B,C,D,E (renamed for this question) ...
0
votes
1answer
34 views

K-Fold cross validating with random forest - how to correctly fit model to every fold?

So I have created K-Folds from my data using this code: ...
2
votes
1answer
34 views

Cross validation and hyperparameter tuning workflow

After reading a lot of articles on cross validation, I am now confused. I know that cross validation is used to get an estimate of model performance and is used to select the best algorithm out of ...
0
votes
0answers
10 views

Are my classification results like accuracy, precision, recall etc significant and valid for general data?

So I have this data let's say of size (2000,11), and I want to do perform a binary classification based on these eleven features. There is a class Imbalance between ...
1
vote
1answer
21 views

Difference between model score on test part and Kaggle public score

I tested my CatBoostModel model on part of data and get 0.92 score, but Kaggle public score was 0.9. I found new hyperparameters via randomsearch, new model score was 0.925, but on Kaggle score fell ...
1
vote
1answer
17 views

Is there any better approach then K folds and nested K folds?

I am trying to understand what problem is K-folds solving. It does not seem to be solving data leakage at all, as we are still testing on test data and then taking an average of all test folds and ...
1
vote
1answer
24 views

Python scikit learn KFold function uneven train, test split

i have the following code below where i have noticed that the length of the train, test split from Kfold.split() is different for the last fold. Any reason why this may be happening and how i can go ...
1
vote
1answer
38 views

Poor binary text classification results

I have a binary NLP classification task to identify text that talks about a target topic from millions of sentences. Between 5-10% of sentences are positive, the rest is negative. I have trained ...
2
votes
0answers
39 views

Cross Validation in Neural Networks

I am training a neural network and doing 10-fold cross validation to measure performance. I have read lots of documentation and forums telling that the set of weights that should be saved or ...
0
votes
1answer
34 views

Is it necessary to use stratified sampling if I am using SMOTE already?

I have already applied SMOTE to my imbalanced dataset with more than 300K observations. Does it still make sense to use stratified K-fold cross validation rather than simply ordinary K-fold cross ...
1
vote
0answers
17 views

How should you preprocess the data before K-fold validation?

I often see Kaggle notebook authors preprocessing the entire training data prior to splitting it for K-fold validation, but does this have a risk of leaking information into the validation set each ...
1
vote
0answers
25 views

How to implement kfold and cv into Hybrid feature selection and evaluate the classification model performance?

I have been working on a Hybrid feature selection combined with hyperopt package for hyperparameter tuning and I am thinking about evaluating the performance of several model classifiers. I looked ...
1
vote
2answers
35 views

Understanding Sklearns learning_curve

I have been using sklearns learning_curve , and there are a few questions I have that are not answered by the documentation(see also here and here), as well as questions that are raised by the ...
0
votes
1answer
36 views

Query regarding surprising spike in accuracy of ML model

I implemented all the major ML models (Logistic Regression, Naive Bayes, SVM, KNN, Decision Tree, Random Forest, Ada Boost & XGBoost) on my dataset. My stratified cross-validation scores are ...
0
votes
1answer
25 views

Comparing accuracies of Grid Search CV & Randomized Search CV with K-Fold Cross Validation?

Are Grid Search CV & Randomized Search CV always/necessarily supposed to give more accurate results after hyperparameter tuning as compared to K-Fold Cross Validation?
0
votes
1answer
33 views

Steps of multiclass classification problem

So this question is more theoretical, than a practical one. I got a dataframe with 4 classes of cars' body types (e.g. sedan, hatchback, etc.) and different characteristics (doors, seats, maximum ...
2
votes
2answers
113 views

Removing the outliers improved my models, is what I did good or bad?

I used cross validation on my data (11000 rows) with maximum salary of 10000 and after some cleaning I got to rmse=70. Then I tried to remove the outliers 10 times just to try things now I have 9000 ...
1
vote
1answer
49 views

Splitting the dataset manually for k-Fold Cross-Validation

I manually divided the dataset into three sets: train, test, and validation. Each set includes several folders, one for each patient. Each patient has many images from a different point of view. As a ...
0
votes
2answers
66 views

I am attempting to implement k-folds cross validation in python3. What is the best way to implement this? Is it preferable to use Pandas or Numpy? [closed]

I am attempting to create a script to implement cross validation in data. However, the splits cannot randomly take any records, so the training and testing can be done on equal data splits for each ...
1
vote
2answers
45 views

How to get best data split from cross validation

I have trained a Random forest regressor which is giving me a rmse score of 70.72. But when I tried the same model in cross_val_score with a cv of 10, it gave me an array that looks something like ...
1
vote
1answer
59 views

Cross-validation split for modelling data with timeseries behavior

Background: I have a dataset that is generated every month (it is similar with card data that contains card demography and transactions every month and new accounts can be added in the middle of data ...
2
votes
1answer
24 views

leave one pair out cross validation

I am trying to train and validate my datasets which contains 17 datasets. I have divided them as 15 for training and 2 for validation. In the process, I train on 15 datasets and use the generated ...
1
vote
1answer
28 views

How can I use the K-fold cross validation for one-class classifier? [closed]

I train a one-class classifier by Class X. In the testing stage, I use Class X and Class Y for validation. I want to compute the F-score metric of Class Y. How can I use the K-fold cross-validation ...
1
vote
2answers
44 views

Why cant we further tune/change the model after evaluating on the test set?

Every thread on stackexchange that I've found says that you can only use the test set once and thats it. So for instance, if you used a linear regression model and got poor results on the test set, ...
0
votes
0answers
37 views

Building a custom scoring function to find mean time-dependent AUC

I’m working on a survival analysis to predict 1-year mortality. I’m trying to build a custom score function that maximizes mean time-dependent AUC. Here is a description of the time-dependent AUC ...
0
votes
0answers
17 views

Qaulity assurance framework for ML/AI

I'd like to hear how organisations are applying a quality framework to ML/AI work. I'm struggling to find good content or good practice on this. By saying quality frameworks, for example, if I was ...
0
votes
1answer
133 views

New to keras neural network and k-fold cross validation

I'm new to learning neural networks and I found an example online to test accuracy with k-fold cross validation. The example is for binary data but I want to test MAE or RMSE (I guess?) for my ...
0
votes
0answers
17 views

Training and validation accuracy stagnating after a few epochs for text embeddings

I have text embeddings (768 dimensional vectors). I tried to build a feed forward neural network on classify the text into two classes. The network I used. ...
0
votes
0answers
30 views

Cross-validation for Random forest

I want to use cross-validation method for Random forest and compare the performance with other algorithms in the same condition. I heard from somebody that Random forest does not need cross-validation....
0
votes
1answer
76 views

Learning curves

I am working on a multiclass classification problem. I want to know whether my model is overfitting or underfitting. I am learning how to plot learning curves and have 4 doubts. 1.) Is the ordering of ...
0
votes
0answers
18 views

LSTM Data Preparation Input Shape

I have a 2-D dataframe df: ...
1
vote
1answer
48 views

Perfect scores for multiclass classification

I am working on a multiclass classification problem with 3 (1, 2, 3) classes being perfectly distributed. (70 instances of each class resulting in (210, 8) dataframe). Now my data has all the 3 ...
0
votes
0answers
22 views

Time-Series Cross-Validation for LSTM

Is it at all possible to separate my data into train/test sets with cross validation for time series data? I am experimenting with a LSTM model. Also, I am hoping to prevent data leakage/peaking in ...
3
votes
3answers
385 views

Model performance worsens after Cross Validation

I am training a logistic regression model on a dataset with only numerical features. I performed the following steps:- 1.) heatmap to remove collinearity between variables 2.) scaling using ...
0
votes
0answers
12 views

Model on non-iid data performing badly

I am on this lecture about non-iid data where we generated a timeseries data using the function below: ...
1
vote
0answers
25 views

Contradictory learning curves in cross validation

I'm fitting gradient boosted decision trees (lightgbm) to model a regression problem. The data is extremely noisy, $R^2 \approx 0$. I'm trying to improve the fitting procedure using 10 fold cross ...
1
vote
1answer
32 views

n_jobs=-1 or n_jobs=1?

I am confused regarding the n_jobs parameter used in some models and for CV. I know it is used for parallel computing, where it includes the number of processors specified in n_jobs parameter. So if I ...
1
vote
1answer
60 views

Decision Tree taking too long to execute

I am training a Decision Tree Regressor on a relatively small data. The dimensions of my train and test sets are (34164, 10) and (8514, 10). Here is the relevant code: ...
0
votes
0answers
72 views

Cross validation in SpaCy NER

I'm working on a custom NER model that I created with SpaCy, and for training/testing purposes I would like to use cross validation. Does SpaCy have the option to somehow perform this? If not, what ...
0
votes
2answers
57 views

aggregation of feature importance

I have more of a conceptual question I was hoping to get some feedback on. I am trying to run a boosted regression ML model to identify a subset of important predictors for some clinical condition. ...
-1
votes
1answer
102 views

nested cross validation vs. train-test split

I am trying to understand the main benefits of conducting a nested cross-validation compared to a simpler train-test split. Let us say I would like to build a prediction model. I initially split my ...
0
votes
0answers
23 views

How do I verify and test a machine learning model against reality during time?

As a software engineers we familiar with a concept of testing (unit, integration, e2e) Tests give us a level of confidence about the code and changes in our code. Looks like for ML the "code"...
0
votes
1answer
50 views

Is Cross validation and GridSearchCV required every time we train a model?

I have a repetitive process that will build a model weekly based on the previous week's data. So while in development I tried GridSearchCV and cross-validation to ...
1
vote
0answers
58 views

How to pass manually split data to cross-validation

I have to perform a binary classification. My dataset is quite small 280 samples and quite imbalanced (1:10 ratio). I kept around 100 sample as testing and about 140 for training. My input variables ...
0
votes
0answers
69 views

Train/Validation/Test split and K-fold Cross Validation

I have a dataset that I have split in train, validation and test subsets. I want to evaluate several CNN architectures and hyperparameters so I have trained several models with different ...
0
votes
1answer
19 views

Can we cross-validate the neuron no. and hidden layer no. of a feed-forward neural network?

Confused about tons of relevant info on the web, so I appreciate if anyone of you can clarify for me. I will use the model to binary classify for your reference. As for the problem, let's say in a ...

1
2 3 4 5
12