Questions tagged [validation]
The validation tag has no usage guidance.
61
questions
0
votes
0
answers
16
views
Why isn't the validation data loss close to the test data loss?
First I set aside about 15% of my data as test data. Then, I used tensorflow.keras to create a relatively simple neural net model. Then I set the model.fit() parameter validation_split=0.2, so 20% of ...
- 1
0
votes
0
answers
20
views
Training with few samples, dropping training loss but constant validation loss
I am training a resnet50-based model using transfer learning. My dataset has 10 classes and about 10 occurrences per class, so it is very small. The training loss is decreasing steadily to 0.07 for ...
0
votes
0
answers
13
views
Do ML model measurements and validation standards (e.g. NIST, ISO) exists for the finance, healthcare, and technology industries? Provide citations
Normally, for example, we talk about splitting datasets into training and test datasets. But. The splitting % per train and test sets happens in a subjective manner. Sometimes. The train is 60% or 70%,...
- 156
0
votes
1
answer
30
views
Is it a problem to use the test dataset for the hyperparameter tuning, when I want to compare 2 classification algorithms on the 10 different dataset?
I know that we should use the validation set to perform hyperparameter tuning and that test dataset is not anymore really the test if it is used for hyperparameter tuning. But is this a problem if i ...
- 1
-1
votes
1
answer
87
views
ARMA model using different train and test/validation datasets
In sklearn I am used to having distinct train and test datasets. In other words, I train a model's parameters on the features from a training set, and then apply ...
- 99
0
votes
0
answers
37
views
Classification Threshold Optimization after GridSearchCV
In my machine learning problem I am using a CNN to classify images.
Since my dataset is imbalanced I want to perform classification probability threshold tuning so I can find the optimal balance ...
0
votes
1
answer
31
views
How to select the validation loss value in this model to be compared with other models?
I'm training an LSTM model. I'm confusing about the validation loss of the model. Which value better represents the validation loss of the model? Is it the last value I obtain in the floowing loop, or ...
0
votes
0
answers
13
views
Scaling datasets for multi-dataset time series
Suppose that I have training data with dimension $(N,H,F)$, where $N$ represents the number of different datasets, $H$ is the history size and $F$ is the input size. Normalizing each dataset over the ...
- 137
0
votes
0
answers
23
views
Final Model Training Problem - Overfitting
I am working on a CNN project for multiclass classification. I implemented hyperparameter optimization to find the most suitable model, during which I got a best accuracy of 97.38%. I then took this ...
- 23
0
votes
1
answer
58
views
Testing RANSAC regression model
I am going to build the model (e.g. multiple linear regression) to predict the appartment cost in my city. First I have to find outliers in training data. For this task RANSAC regression algorithm ...
0
votes
0
answers
30
views
Internal Validation of K-Modes in R
I am wondering if anyone knows of an R library suitable for internal validation of the k-modes algorithm?
Therefore, an R library with silhouette coefficient and other indices suitable for validating ...
- 203
0
votes
1
answer
24
views
Train/val/test approach for hyperparameter tuning
When looking to train a model, does it make sense to have a 60-20-20 train val test split, first hyper parameter tuning over the training dataset, using the validation set, picking the best model. ...
- 91
0
votes
0
answers
21
views
Can not validate LSTM/GRU if return_sequences=True
As I understand the last LSTM before output layer should be set to
model.add(LSTM(lstmUnits, return_sequences=False))
but in such a case I can not validate while ...
- 101
1
vote
1
answer
91
views
Validation accuracy almost zero while training accuracy much higher
I am training a large dataset with CNN. The training dataset has 46182 samples and 1414 different classes , while the validation dataset has 14053 samples and 790 different classes.
The first question ...
0
votes
0
answers
38
views
Large change in validation loss, small change in training loss
I'm training a multi-task, multi-label neural network. I am attempting to tune the architecture and am having some trouble interpreting the learning curves. Particularly, when I look at the learning ...
- 63
2
votes
1
answer
687
views
Validation and training loss of a model are not stable
Below I have a model trained and the loss of both the training dataset (blue) and validation dataset (orange) are shown. From my understanding, the ideal case is that both validation and training loss ...
- 231
1
vote
2
answers
36
views
why it would be improper to compute and use test set means?
I have 2 questions regarding the whole subject of the data set in machine learning and I would be happy to receive an answer :)
1.Why it would be improper to compute and use test set means and ...
- 9
2
votes
1
answer
329
views
How to build a model when we have three separate train, validation, and test sets?
I have a data set which should be divided into train, test, and validation sets.
...
- 1,297
1
vote
0
answers
30
views
Neural net patience moving average
To my understanding, when we use patience = 8 when training a neural net, if there is no improvement on the loss (usually the validation set loss) for 8 epochs, then we would stop the training and ...
- 11
1
vote
1
answer
27
views
In cross validation, should the test dataset not be fixed
Would this work?
We want to train a neural net.
We have 50 datapoints and want a split of 30 for train, 10 for validation, 10 for test.
We want to do 5-fold cross validation.
We use the following ...
- 11
1
vote
0
answers
39
views
How to perform bootstrap validation on CART decision tree?
I have a relatively small dataset n = 500 for which I am training a CART decision tree.
My dataset has about 30 variables and the outcome has 3 classes.
I am using CART for interpretability purposes, ...
- 11
0
votes
1
answer
25
views
How to proceed when training data change frequently (in production)?
I'm working with a Recommendation System that would take as parameters a bunch of "tweets" a user see during his navegation on a mobile app. Every tweet has a property, like a category (...
1
vote
0
answers
17
views
What to do if your adversarial validation show different distributions for an NLP problem?
I was trying to figure out if the test set from a competition is similar to the train set. This was done in a NLP competition, in which I had two columns, tweet and type, and I needed to predict the ...
- 191
0
votes
1
answer
665
views
Training loss decreasing while Validation loss is not decreasing
I am wondering why validation loss of this regression problem is not decreasing while I have implemented several methods such as making the model simpler, adding early stopping, various learning rates,...
1
vote
1
answer
78
views
Model Performance on external validation Set really low?
I am using the LGBM model for binary classification. My train and test accuracies are 87% & 82% respectively with cross-validation of 89%. ROC-AUC score of 81%. But when evaluating model ...
- 47
0
votes
1
answer
132
views
What does it mean if the validation accuracy is equal to the testing accuracy?
I am training a CNN model for my specific problem. I have divided the dataset into 70% training set, 20% validation set, and 10% test set. The validation accuracy achieved was 95% and the test ...
- 145
1
vote
0
answers
26
views
How can I calculate de AUC PR of my classifiers in a multiclass scenario?
I'm developing image classifiers in a context with 25k images and 50 classes. The dataset is imbalanced.
Some papers recommend AUC PR for comparing the performance of my classifiers in this setting. ...
- 139
0
votes
1
answer
132
views
Training Loss or Validation Loss for Hyperparameter Optimisation
When performing HO, should I be looking to train each model (each with different hyperparameter values, e.g. with RandomSearch picking those values) on the training data, and then the best one is ...
- 91
1
vote
0
answers
22
views
Is Callback / early stopping and validation set is not mandatory
I just noticed that in mostly github repositry of research papers they didnt implemented early stopping criteria and they didnt use validation set but whats the reason behind this?
- 171
0
votes
1
answer
23
views
How to print the corresponding c of the lowest classification error on the validation data
I'm currently measuring the overall classification error for an SVM classifier and I'm varying the regularization value C.
In the following code, how can I print in ...
1
vote
1
answer
1k
views
0
votes
3
answers
168
views
How to do modelling for pairs of non i.i.d. data?
I have a dataset in which i have the labels for candidates on whether they would be hired,interviewed_and_failed,not_interviewed_at_all. The task is to predict for new jobs/new candidates what these ...
- 73
2
votes
3
answers
370
views
What is exactly the difference between Validation data and Testing data
I asked this question on stack overflow and was told that this is a better place for it.
I am confused with the terms validation and testing, is validating the model same as testing it? is it possible ...
- 23
1
vote
2
answers
114
views
Updating a train/val/test set
It is considered best practice to split your data into a train and test set at the start of a data science / machine learnign project (and then your train set further into a validation set for ...
- 448
0
votes
0
answers
30
views
Validation accuracy have an almost good accuracy but loss function is high too
Based on my project, I find a little problem there with the statement like this.
I want to make model with neural networks for text dataset. Then I use Pad Sequence for my text and Array for the ...
- 13
2
votes
2
answers
311
views
Dataset and why use evaluate()?
I am starting in Machine Learning, and I have doubts about some concepts. I've read we need to split our dataset into training, validation and test sets. I'll ask four questions related to them.
1 - ...
- 125
0
votes
1
answer
568
views
Logarithmic scale for a learning curve [closed]
I'm plotting the learning curve with Python with the following code:
...
- 111
1
vote
1
answer
116
views
Difference between model score on test part and Kaggle public score
I tested my CatBoostModel model on part of data and get 0.92 score, but Kaggle public score was 0.9. I found new hyperparameters via randomsearch, new model score was 0.925, but on Kaggle score fell ...
- 125
1
vote
0
answers
19
views
Different data dimension when indexing with R
I am facing on something that I cannot explain, so may be someone could explain me. I have a dataset which contain 102011 data.
I want to take 70% for the train and 30% remaining for the validation. I ...
- 137
1
vote
3
answers
389
views
Spliting Training Test and Validation for Image Dataset
I have 600 images in the training folder, 200 images in the validation folder, and 200 images in the test folder. Suppose if I fit the training data generator and validation data generator for some ...
- 36
3
votes
1
answer
2k
views
Does validation data has any effect on training or it acts solely without affecting the training?
When using Keras library of Python, we use validation data with training data while training our model. In every epoch, we get a validation accuracy. Does this validation accuracy have any effect on ...
- 133
1
vote
1
answer
36
views
Measure performance of classification model for training on different snapshots
I am trying to do binary classification on some chronological data. Let's assume we have weekly data from the first week of 2017 through the last week of 2020. Now we have found out that 26 weeks of ...
- 189
1
vote
1
answer
62
views
Using Z-test score to evaluate model performance
I think I know the answer to this question but I am looking for a sanity check here: Is it appropriate to use z-test scores in order to evaluate the performance of my model?
I have a binary model that ...
- 2,079
0
votes
1
answer
170
views
Does it make sense to repeat calculating AUC in logistic regression?
I have a question regarding logistic regression models and testing its skill.
I am not quite sure if I understand correctly how the ROC Curve is established.
When calculating the ROC curve, is a train ...
- 15
2
votes
0
answers
219
views
ValueError: Input contains NaN, infinity or a value too large for dtype('float32'). Any advice? [closed]
The error below presented itself when attempting to assemble a PCA.
My code:
...
- 21
0
votes
1
answer
490
views
Keras: Very high loss for Autoencoder
I am trying to implement an autoencoder for prediction of multiple labels using Keras. This is a snippet:
...
- 101
0
votes
0
answers
61
views
How many epochs should all the data be trained on after training with validation finds when validation and training diverge?
One uses and train/test split to use their training data to get an idea of how many epochs to train with. If the validation accuracy starts going down while the training accuracy is still going up, ...
- 101
1
vote
2
answers
757
views
dataset split for image classification
I am trying to do image classification for 14 categories (around 1000 images for each cat). And i initially created two folders for training and validation. In this case, do I still need to set a ...
0
votes
0
answers
85
views
Validation accuracy does not increase for binary classification using GNN
I am trying to perform graph classification with GINConv model of GNN. I have tried everything from varying dropouts to weight decay (for L2 regularization), learning rate (1e-6 to 1e-3), batch ...
2
votes
0
answers
15
views
Validation set after hyperparameter tuning [duplicate]
Let's say I'm comparing few models, and for my dataset I'm using train/validation/test split, and not cross validation. Let's say I'm completely done with parameter tuning for one of them and want to ...
- 21