Questions tagged [validation]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

Training loss decreasing while Validation loss is not decreasing

I am wondering why validation loss of this regression problem is not decreasing while I have implemented several methods such as making the model simpler, adding early stopping, various learning rates,...
user avatar
1 vote
1 answer
25 views

Model Performance on external validation Set really low?

I am using the LGBM model for binary classification. My train and test accuracies are 87% & 82% respectively with cross-validation of 89%. ROC-AUC score of 81%. But when evaluating model ...
user avatar
  • 27
0 votes
0 answers
6 views

Why is my validation loss never INcreasing?

I am currently training different neural networks for the binary classification of images. When using the logistic regression, my validation loss never increases, even not after 5000 epochs. I thought ...
user avatar
0 votes
1 answer
31 views

What does it mean if the validation accuracy is equal to the testing accuracy?

I am training a CNN model for my specific problem. I have divided the dataset into 70% training set, 20% validation set, and 10% test set. The validation accuracy achieved was 95% and the test ...
user avatar
  • 13
1 vote
0 answers
13 views

How can I calculate de AUC PR of my classifiers in a multiclass scenario?

I'm developing image classifiers in a context with 25k images and 50 classes. The dataset is imbalanced. Some papers recommend AUC PR for comparing the performance of my classifiers in this setting. ...
user avatar
  • 127
0 votes
1 answer
21 views

Training Loss or Validation Loss for Hyperparameter Optimisation

When performing HO, should I be looking to train each model (each with different hyperparameter values, e.g. with RandomSearch picking those values) on the training data, and then the best one is ...
user avatar
  • 21
1 vote
0 answers
13 views

Is Callback / early stopping and validation set is not mandatory

I just noticed that in mostly github repositry of research papers they didnt implemented early stopping criteria and they didnt use validation set but whats the reason behind this?
user avatar
  • 175
0 votes
1 answer
20 views

How to print the corresponding c of the lowest classification error on the validation data

I'm currently measuring the overall classification error for an SVM classifier and I'm varying the regularization value C. In the following code, how can I print in ...
user avatar
1 vote
1 answer
32 views

Why am I getting different prediction result after every run?

I have a simple lstm model ...
user avatar
0 votes
0 answers
10 views

StratifiedKFold in Pixel-Wise Image Segmentation

When I use regular code for StratifiedKFold Cross-validation ...
user avatar
  • 103
0 votes
3 answers
100 views

How to do modelling for pairs of non i.i.d. data?

I have a dataset in which i have the labels for candidates on whether they would be hired,interviewed_and_failed,not_interviewed_at_all. The task is to predict for new jobs/new candidates what these ...
user avatar
0 votes
0 answers
19 views

Is it okay to use test data to predict next record?

If I have say 1 year daily close price of a stock and I divide it in ratio of 80:20 as train:test data. Now I use TimeSeriesGenerator to fit the model on train data. After fitting the model I want to ...
user avatar
2 votes
3 answers
106 views

What is exactly the difference between Validation data and Testing data

I asked this question on stack overflow and was told that this is a better place for it. I am confused with the terms validation and testing, is validating the model same as testing it? is it possible ...
user avatar
  • 23
1 vote
2 answers
36 views

Updating a train/val/test set

It is considered best practice to split your data into a train and test set at the start of a data science / machine learnign project (and then your train set further into a validation set for ...
user avatar
  • 358
0 votes
0 answers
16 views

Validation accuracy have an almost good accuracy but loss function is high too

Based on my project, I find a little problem there with the statement like this. I want to make model with neural networks for text dataset. Then I use Pad Sequence for my text and Array for the ...
user avatar
  • 13
2 votes
2 answers
151 views

Dataset and why use evaluate()?

I am starting in Machine Learning, and I have doubts about some concepts. I've read we need to split our dataset into training, validation and test sets. I'll ask four questions related to them. 1 - ...
user avatar
  • 125
0 votes
1 answer
66 views

Logarithmic scale for a learning curve [closed]

I'm plotting the learning curve with Python with the following code: ...
user avatar
  • 11
1 vote
1 answer
38 views

Difference between model score on test part and Kaggle public score

I tested my CatBoostModel model on part of data and get 0.92 score, but Kaggle public score was 0.9. I found new hyperparameters via randomsearch, new model score was 0.925, but on Kaggle score fell ...
user avatar
0 votes
0 answers
15 views

After finalizing the hyper-parameters and model using a val set, is it recommended to train the final model using both train and validation sets?

Assume I have all the 3 pre-standardized splits of a dataset (train, val, set), with groundtruth for all 3. Usually many datasets are of this form. Once I have finalized the best configuration of ...
user avatar
  • 131
0 votes
0 answers
22 views

validation loss early increase (during warm-up)

Several questions have been asked about validation loss behavior during training of a DNN. It's clear to me that validation loss and accuracy are somehow correlated, but their curves can differ from ...
user avatar
  • 123
1 vote
0 answers
19 views

Different data dimension when indexing with R

I am facing on something that I cannot explain, so may be someone could explain me. I have a dataset which contain 102011 data. I want to take 70% for the train and 30% remaining for the validation. I ...
user avatar
0 votes
0 answers
8 views

Split data using existing model's performance

For splitting of the data into train/test/val I use stratified sampling. Then I confirm that metadata distributions represent the original dataset well enough. I want to start considering error of an ...
user avatar
0 votes
0 answers
7 views

What is the validation strategy for approximate string search?

I am working a approximate string search algorithm. I am wondering how to go with validating that an algorithm is better than another. I can not come up with validation set, since I have no example of ...
user avatar
  • 201
0 votes
0 answers
13 views

Any books or resources about how to approach "purely synthethic expressions" of physical phenomena?

Over and over again I come to think that "it's cumbersome to collect empirical data". Yet it's often viewed as a necessity for explaining empirical phenomena. But then I idealize that: It ...
user avatar
  • 379
0 votes
0 answers
14 views

Validation output in a custom training loop not working - Tensorflow

I am new to Deep Learning and I am trying to learn more about implementation in Tensorflow and Keras. I am basing my work on this link : https://www.tensorflow.org/guide/keras/...
user avatar
  • 1
1 vote
2 answers
56 views

Spliting Training Test and Validation for Image Dataset

I have 600 images in the training folder, 200 images in the validation folder, and 200 images in the test folder. Suppose if I fit the training data generator and validation data generator for some ...
user avatar
  • 36
0 votes
0 answers
17 views

Should training and validation patches come from different images/files for image segmentation

In the situation where we have an image segmentation problem and we feed patches (smaller parts of the images) to the model, should the training and validation patches come from different files (...
user avatar
0 votes
1 answer
154 views

Does validation data has any effect on training or it acts solely without affecting the training?

When using Keras library of Python, we use validation data with training data while training our model. In every epoch, we get a validation accuracy. Does this validation accuracy have any effect on ...
user avatar
1 vote
1 answer
18 views

Measure performance of classification model for training on different snapshots

I am trying to do binary classification on some chronological data. Let's assume we have weekly data from the first week of 2017 through the last week of 2020. Now we have found out that 26 weeks of ...
user avatar
  • 179
1 vote
1 answer
20 views

Using Z-test score to evaluate model performance

I think I know the answer to this question but I am looking for a sanity check here: Is it appropriate to use z-test scores in order to evaluate the performance of my model? I have a binary model that ...
user avatar
0 votes
0 answers
62 views

Python 3.9 - EEG Signal Classification for P300 Speller BCI - Validation loss not reducing below 40%

I am working with a classic P300 Speller BCI. I am trying to create a classification model for classifying EEG signal epochs into two categories, "Target" and "Non-Target". The ...
user avatar
0 votes
1 answer
72 views

Does it make sense to repeat calculating AUC in logistic regression?

I have a question regarding logistic regression models and testing its skill. I am not quite sure if I understand correctly how the ROC Curve is established. When calculating the ROC curve, is a train ...
user avatar
2 votes
0 answers
83 views

ValueError: Input contains NaN, infinity or a value too large for dtype('float32'). Any advice? [closed]

The error below presented itself when attempting to assemble a PCA. My code: ...
user avatar
0 votes
1 answer
142 views

Keras: Very high loss for Autoencoder

I am trying to implement an autoencoder for prediction of multiple labels using Keras. This is a snippet: ...
user avatar
0 votes
0 answers
44 views

How many epochs should all the data be trained on after training with validation finds when validation and training diverge?

One uses and train/test split to use their training data to get an idea of how many epochs to train with. If the validation accuracy starts going down while the training accuracy is still going up, ...
user avatar
1 vote
2 answers
208 views

dataset split for image classification

I am trying to do image classification for 14 categories (around 1000 images for each cat). And i initially created two folders for training and validation. In this case, do I still need to set a ...
user avatar
0 votes
0 answers
47 views

Validation accuracy does not increase for binary classification using GNN

I am trying to perform graph classification with GINConv model of GNN. I have tried everything from varying dropouts to weight decay (for L2 regularization), learning rate (1e-6 to 1e-3), batch ...
user avatar
2 votes
0 answers
15 views

Validation set after hyperparameter tuning [duplicate]

Let's say I'm comparing few models, and for my dataset I'm using train/validation/test split, and not cross validation. Let's say I'm completely done with parameter tuning for one of them and want to ...
user avatar
  • 21
0 votes
1 answer
62 views

How to compare a machine learning model and a rule based model

The difference between machine leaning models and rule based model is that you feed input and output to machine learning models to get rules where you feed input and rules to rule based models to get ...
user avatar
0 votes
1 answer
1k views

How is the validation set processed in PyTorch?

Say, one uses the MNIST dataset and splits the provided training data of size 60,000 into a training set (50,000) and a validation set (10,000). The provided test data of size 10,000 is used as the ...
user avatar
1 vote
1 answer
101 views

Does overfitting depend only on validation loss or both training and validation loss?

There are several scenarios that can occur while training and validating: Both training loss and validation loss are decreasing, with the training loss lower than the validation loss. Both training ...
user avatar
  • 131
2 votes
1 answer
60 views

Using the whole dataset for testing (not validation) in case of small datasets

for an object detection task I created a small dataset to train an object detector. The class frequency is more or less balanced, however I defined some additional attributes with environmental ...
user avatar
3 votes
1 answer
130 views

Why do machine learning engineers insist on training with more data than validation set?

Among my colleagues I have noticed a curious insistence on training with, say, 70% or 80% of data and validating on the remainder. The reason it is curious to me is the lack of any theoretical ...
user avatar
1 vote
1 answer
75 views

Is it right to maintain the train distribution in test set for unbalanced data?

If the training set was unbalanced the chances are the model will be biased. But if the data distribution in the test set is the same distribution as the train set, this kind of bias is not going to ...
user avatar
0 votes
0 answers
122 views

How can realize the evaluation/validation of unsupervised models through unlabeled data?

I'm researching anomaly detection, which is nothing else than outliers detection on a set of time-series web servers access log data or network traffic. Recently I re-faced to following fundamental ...
user avatar
  • 369
0 votes
2 answers
194 views

Validation loss and validation accuracy stay the same in NN model

I am trying to train a keras NN regression model for music emotion prediction from audio features. (I am a beginner in NN and I am doing this as study project.) I have 193 features for training/...
user avatar
  • 1
0 votes
0 answers
65 views

How to set thresholds to automate the management of model drift?

I have a trained ML Model and new data coming in every week. The new data sometimes vary too much (in a statistical sense) and, therefore, the performance of the ML model degrades. The data-set schema ...
user avatar
0 votes
1 answer
53 views

Augmentation on test dataset and validation dataset

I'm training a segmentation model (computer-vision). Thus, my dataset contains images and masks (binary segmentation of objects). I'm augmenting the training dataset (applying random crop, rotation or ...
user avatar
  • 114
0 votes
1 answer
413 views

SMOTE train test split with validation data [duplicate]

Would like to ask, in which way to use SMOTE? My dataset is imbalanced and a multiclass problem. As I read in many posts, use SMOTE method only for the training dataset (X_train and y_train). Not for ...
user avatar
  • 329