Questions tagged [cross-validation]

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy.

Filter by
Sorted by
Tagged with
1
vote
2answers
33 views

How to get best data split from cross validation

I have trained a Random forest regressor which is giving me a rmse score of 70.72. But when I tried the same model in cross_val_score with a cv of 10, it gave me an array that looks something like ...
1
vote
1answer
38 views

Cross-validation split for modelling data with timeseries behavior

Background: I have a dataset that is generated every month (it is similar with card data that contains card demography and transactions every month and new accounts can be added in the middle of data ...
2
votes
1answer
16 views

leave one pair out cross validation

I am trying to train and validate my datasets which contains 17 datasets. I have divided them as 15 for training and 2 for validation. In the process, I train on 15 datasets and use the generated ...
1
vote
1answer
26 views

How can I use the K-fold cross validation for one-class classifier? [closed]

I train a one-class classifier by Class X. In the testing stage, I use Class X and Class Y for validation. I want to compute the F-score metric of Class Y. How can I use the K-fold cross-validation ...
1
vote
2answers
39 views

Why cant we further tune/change the model after evaluating on the test set?

Every thread on stackexchange that I've found says that you can only use the test set once and thats it. So for instance, if you used a linear regression model and got poor results on the test set, ...
0
votes
0answers
23 views

Building a custom scoring function to find mean time-dependent AUC

I’m working on a survival analysis to predict 1-year mortality. I’m trying to build a custom score function that maximizes mean time-dependent AUC. Here is a description of the time-dependent AUC ...
0
votes
0answers
14 views

Qaulity assurance framework for ML/AI

I'd like to hear how organisations are applying a quality framework to ML/AI work. I'm struggling to find good content or good practice on this. By saying quality frameworks, for example, if I was ...
0
votes
1answer
27 views

New to keras neural network and k-fold cross validation

I'm new to learning neural networks and I found an example online to test accuracy with k-fold cross validation. The example is for binary data but I want to test MAE or RMSE (I guess?) for my ...
0
votes
0answers
15 views

Training and validation accuracy stagnating after a few epochs for text embeddings

I have text embeddings (768 dimensional vectors). I tried to build a feed forward neural network on classify the text into two classes. The network I used. ...
0
votes
0answers
21 views

Cross-validation for Random forest

I want to use cross-validation method for Random forest and compare the performance with other algorithms in the same condition. I heard from somebody that Random forest does not need cross-validation....
0
votes
1answer
45 views

Learning curves

I am working on a multiclass classification problem. I want to know weather my model is overfitting or underfitting. I am learning how to plot learning curves and have 4 doubts. 1.) Is the ordering of ...
0
votes
0answers
14 views

LSTM Data Preparation Input Shape

I have a 2-D dataframe df: ...
1
vote
1answer
41 views

Perfect scores for multiclass classification

I am working on a multiclass classification problem with 3 (1, 2, 3) classes being perfectly distributed. (70 instances of each class resulting in (210, 8) dataframe). Now my data has all the 3 ...
0
votes
0answers
12 views

Time-Series Cross-Validation for LSTM

Is it at all possible to separate my data into train/test sets with cross validation for time series data? I am experimenting with a LSTM model. Also, I am hoping to prevent data leakage/peaking in ...
3
votes
3answers
372 views

Model performance worsens after Cross Validation

I am training a logistic regression model on a dataset with only numerical features. I performed the following steps:- 1.) heatmap to remove collinearity between variables 2.) scaling using ...
0
votes
0answers
12 views

Model on non-iid data performing badly

I am on this lecture about non-iid data where we generated a timeseries data using the function below: ...
1
vote
0answers
24 views

Contradictory learning curves in cross validation

I'm fitting gradient boosted decision trees (lightgbm) to model a regression problem. The data is extremely noisy, $R^2 \approx 0$. I'm trying to improve the fitting procedure using 10 fold cross ...
1
vote
1answer
30 views

n_jobs=-1 or n_jobs=1?

I am confused regarding the n_jobs parameter used in some models and for CV. I know it is used for parallel computing, where it includes the number of processors specified in n_jobs parameter. So if I ...
0
votes
0answers
33 views

Decision Tree taking too long to execute

I am training a Decision Tree Regressor on a relatively small data. The dimensions of my train and test sets are (34164, 10) and (8514, 10). Here is the relevant code: ...
0
votes
0answers
43 views

Cross validation in SpaCy NER

I'm working on a custom NER model that I created with SpaCy, and for training/testing purposes I would like to use cross validation. Does SpaCy have the option to somehow perform this? If not, what ...
0
votes
2answers
25 views

aggregation of feature importance

I have more of a conceptual question I was hoping to get some feedback on. I am trying to run a boosted regression ML model to identify a subset of important predictors for some clinical condition. ...
-1
votes
1answer
43 views

nested cross validation vs. train-test split

I am trying to understand the main benefits of conducting a nested cross-validation compared to a simpler train-test split. Let us say I would like to build a prediction model. I initially split my ...
0
votes
0answers
21 views

How do I verify and test a machine learning model against reality during time?

As a software engineers we familiar with a concept of testing (unit, integration, e2e) Tests give us a level of confidence about the code and changes in our code. Looks like for ML the "code"...
0
votes
1answer
36 views

Is Cross validation and GridSearchCV required every time we train a model?

I have a repetitive process that will build a model weekly based on the previous week's data. So while in development I tried GridSearchCV and cross-validation to ...
1
vote
0answers
32 views

How to pass manually split data to cross-validation

I have to perform a binary classification. My dataset is quite small 280 samples and quite imbalanced (1:10 ratio). I kept around 100 sample as testing and about 140 for training. My input variables ...
0
votes
0answers
55 views

Train/Validation/Test split and K-fold Cross Validation

I have a dataset that I have split in train, validation and test subsets. I want to evaluate several CNN architectures and hyperparameters so I have trained several models with different ...
0
votes
1answer
18 views

Can we cross-validate the neuron no. and hidden layer no. of a feed-forward neural network?

Confused about tons of relevant info on the web, so I appreciate if anyone of you can clarify for me. I will use the model to binary classify for your reference. As for the problem, let's say in a ...
1
vote
1answer
38 views

Why do we need cross validation set? [closed]

I know we need to test our model on onseen data, but isn't that test set are for? Also what will happen if we increase K value in kfold?
0
votes
0answers
6 views

Prediction count seems to be way in logistic regression despite good accuracy, precision and recall score

I built a Logistic Regression Model to predict Loan Acceptors. The dataset is 94% non acceptors and 6% acceptors. I've run several logistic regression models one with the original dataset, one after ...
0
votes
0answers
22 views

How to perform cross validation with ktrain

Is there a valid code to perform cross validation with ktrain models ?
0
votes
1answer
22 views

Criteria for saving best model during training neural network?

I am doing 4-class semantic segmentation with U-net using generalised dice loss as loss function. General approach to save best model during training is to monitor validation loss at each epoch and ...
2
votes
1answer
40 views

Training a model with Cross-Validation

I'm training a model with CV and then I'm testing the predictions on a new test set. Am I doing the right thing or is it necessary to test the predictions on a new dataset using Cross-Validation? ...
0
votes
0answers
21 views

Different learning curves on each run

Sorry for the wrong terminology I might use, since I’m a noob. For my supervised learning project for the university I have a dataset (features and labels) which has to evaluated in several ways and ...
0
votes
1answer
498 views

neg_mean_squared_error in cross_val_score [closed]

The string "mean_squared_error" appears to be deprecated in cross_val_score now, and it's saying to use ...
0
votes
0answers
11 views

Understanding the process for generating Random Forest model in caret

This is not so much a problem, as it is me making sure I understand what's happening with my Random Forest algorithm. Below, I've set a few parameters. Am I right in thinking that this is the stages: ...
0
votes
0answers
24 views

Is there an point to using LassoCV with `cross_val_score`

I've seen some jupyter notebooks that seem to combine LassoCV with cross_val_score, and I'm confused what the point is. Usually ...
0
votes
0answers
39 views

gridsearchcv vs LassoCV/RidgeCV in Python

Is there any advantage to using GridSearchCV over LassoCV/RidgeCV? I don't actually understand the point of the former, when we ...
1
vote
2answers
38 views

Cross Validation and bias relation

I found a question (Question 7) here: Question: For k cross-validation, larger k value implies more bias Options: True or False My answer is: True. Reason: Larger K means more folds means smaller ...
1
vote
1answer
27 views

Large variation in cross-validation scores [closed]

I'm training a CNN with 5-fold stratified cross-validation. On the first fold, my accuracy is ~80%, on each subsequent fold the accuracy is ~50%. Finally, upon fitting the entire training set my ...
1
vote
0answers
45 views

Imbalanced dataset, finding the statistical significance of a Matthews Correlation Coefficient (MCC) in binary classification (what is a good MCC)?

I have a very imbalanced dataset. Thus, I am using MCC to evaluate the performance of various ML algorithms. It appears that literature is entirely lacking in ways to evaluate how good an MCC score is....
1
vote
2answers
15 views

Does adding a model complexity penalty to the loss function allow you to skip cross-validation?

It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of ...
3
votes
1answer
90 views

KFold cross validation ambiguity

I just studied K-Fold cross validation technique for finding model parameters and something seemed to be very confusing. Every tutorial I follow says that for K-Fold validation, the whole dataset will ...
0
votes
0answers
13 views

Calculate euclidean distances for KNN and cross validation given a 99x16 10 folds

I'm trying to implement KNN classification with cross-validation implementation in python. The data consists of 10 folds of size 99x64, each with their corresponding label of size 99x1. Do I have to ...
1
vote
0answers
15 views

Which possible models can I use to predict results from this dataset? [closed]

I'm new at predicting things like this. I have a data set. The head of it is shown here: I have yet another data set for the upcoming basketball games that are taking place soon with the score ...
1
vote
1answer
58 views

Using Sklearn's predefined split

I am working on a binary classification task using SVM. The dataset is quite large so I don't want to use k-fold CV for parameter tuning, but instead a simple train-validation-test split. I have done ...
0
votes
0answers
33 views

Can we used both cross validation/nested cross validation technique and early stopping with patient at the same time?

Can we use both cross validation/nested cross validation technique and early stopping with patient at the same time? Using early stopping for each (training, validation) fold and get best result of ...
4
votes
1answer
206 views

How can precision be less than one in Leave-One-Subject-Out binary classification if each subject contains only one class

Say I'm trying to classify a medical condition. Theres only two classes: Sick and Healthy. I build a model and I can't split the data because I don't want data from the same patient being in training ...
1
vote
1answer
28 views

Why is the optimal C chosen by GridSearchCV so small?

I'm trying to use GridSearchCV to select the optimal C value in this simple SVM problem with non-separable samples. The issue I'm having is that when I run the code the optimal C is chosen to be ...
1
vote
1answer
856 views

Why GridSearchCV returns nan?

I am using gridsearchcv to tune the parameters of my model and I also use pipeline and cross-validation. When I run the model to tune the parameter of XGBoost, it returns ...
0
votes
0answers
187 views

How to Determine Alpha values in Random Forest Classifier

I am working on a classification problem where I am applying various machine learning models. I have used DecisionTreeClassifier from Sklearn on my dataset using the following steps: Calculated alpha ...

1
2 3 4 5
11