Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [validation]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
22 views

Is balancing imbalanced validation set for retraining model after hyperparameter tuning required?

The following are basic steps to modelling, but would like to ask in the case of imbalanced data, is balancing of train dataset required when retraining model on train + validation set after ...
curious-24-7's user avatar
1 vote
1 answer
86 views

I dont understand this way of having a stable train/test split even after updating the dataset

...
samsamradas's user avatar
0 votes
0 answers
20 views

Choosing a cluster validation measure for graph clustering algorithm

I am currently solving a clustering problem. Objects to be clustered are represented as sparse vectors in R^N, N=10. The number of objects is about 1kk. To cluster, I build a graph keeping the largest ...
Sergey Tkachenko's user avatar
1 vote
1 answer
21 views

Hyperparameter tuning

Jane trains three different classifiers: Logistic Regression, Decision Tree, and Support Vector Machines on the training set. Each classifier has one hyper-parameter (regularisation parameter, depth-...
Tom's user avatar
  • 11
1 vote
1 answer
95 views

Optimal Number of Epochs for Training Transformer Network on Time series data? Early Stopping and Model Selection Strategies

I have a transformer network that is trained on time series data. The task is to predict if a variable will increase a certain percentage in the next 7 days. The input is data from the 90 previous ...
QCQCQC's user avatar
  • 113
0 votes
2 answers
31 views

how to fix my increasing validation loss and decreasing training loss?

here is the code that got me this, please i need an advise on what to do to correct this. ...
Michael Oyeboade's user avatar
0 votes
0 answers
20 views

Why isn't the validation data loss close to the test data loss?

First I set aside about 15% of my data as test data. Then, I used tensorflow.keras to create a relatively simple neural net model. Then I set the model.fit() parameter validation_split=0.2, so 20% of ...
Alex's user avatar
  • 1
0 votes
0 answers
22 views

Training with few samples, dropping training loss but constant validation loss

I am training a resnet50-based model using transfer learning. My dataset has 10 classes and about 10 occurrences per class, so it is very small. The training loss is decreasing steadily to 0.07 for ...
ml_nnoobb's user avatar
0 votes
0 answers
15 views

Do ML model measurements and validation standards (e.g. NIST, ISO) exists for the finance, healthcare, and technology industries? Provide citations

Normally, for example, we talk about splitting datasets into training and test datasets. But. The splitting % per train and test sets happens in a subjective manner. Sometimes. The train is 60% or 70%,...
Full Array's user avatar
0 votes
1 answer
41 views

Is it a problem to use the test dataset for the hyperparameter tuning, when I want to compare 2 classification algorithms on the 10 different dataset?

I know that we should use the validation set to perform hyperparameter tuning and that test dataset is not anymore really the test if it is used for hyperparameter tuning. But is this a problem if i ...
John B's user avatar
  • 1
0 votes
0 answers
82 views

Classification Threshold Optimization after GridSearchCV

In my machine learning problem I am using a CNN to classify images. Since my dataset is imbalanced I want to perform classification probability threshold tuning so I can find the optimal balance ...
Throwaway123's user avatar
0 votes
1 answer
55 views

How to select the validation loss value in this model to be compared with other models?

I'm training an LSTM model. I'm confusing about the validation loss of the model. Which value better represents the validation loss of the model? Is it the last value I obtain in the floowing loop, or ...
Mouna Ahmen's user avatar
0 votes
1 answer
85 views

Testing RANSAC regression model

I am going to build the model (e.g. multiple linear regression) to predict the appartment cost in my city. First I have to find outliers in training data. For this task RANSAC regression algorithm ...
Irina Svist's user avatar
0 votes
1 answer
34 views

Train/val/test approach for hyperparameter tuning

When looking to train a model, does it make sense to have a 60-20-20 train val test split, first hyper parameter tuning over the training dataset, using the validation set, picking the best model. ...
Socorro's user avatar
  • 111
1 vote
1 answer
166 views

Validation accuracy almost zero while training accuracy much higher

I am training a large dataset with CNN. The training dataset has 46182 samples and 1414 different classes , while the validation dataset has 14053 samples and 790 different classes. The first question ...
roberto bruzzese's user avatar
0 votes
0 answers
48 views

Large change in validation loss, small change in training loss

I'm training a multi-task, multi-label neural network. I am attempting to tune the architecture and am having some trouble interpreting the learning curves. Particularly, when I look at the learning ...
tensormoby's user avatar
2 votes
1 answer
1k views

Validation and training loss of a model are not stable

Below I have a model trained and the loss of both the training dataset (blue) and validation dataset (orange) are shown. From my understanding, the ideal case is that both validation and training loss ...
Avv's user avatar
  • 231
1 vote
2 answers
38 views

why it would be improper to compute and use test set means?

I have 2 questions regarding the whole subject of the data set in machine learning and I would be happy to receive an answer :) 1.Why it would be improper to compute and use test set means and ...
yuvi's user avatar
  • 9
2 votes
1 answer
351 views

How to build a model when we have three separate train, validation, and test sets?

I have a data set which should be divided into train, test, and validation sets. ...
ebrahimi's user avatar
  • 1,307
1 vote
0 answers
35 views

Neural net patience moving average

To my understanding, when we use patience = 8 when training a neural net, if there is no improvement on the loss (usually the validation set loss) for 8 epochs, then we would stop the training and ...
Callum's user avatar
  • 11
1 vote
1 answer
28 views

In cross validation, should the test dataset not be fixed

Would this work? We want to train a neural net. We have 50 datapoints and want a split of 30 for train, 10 for validation, 10 for test. We want to do 5-fold cross validation. We use the following ...
Becca's user avatar
  • 11
1 vote
0 answers
61 views

How to perform bootstrap validation on CART decision tree?

I have a relatively small dataset n = 500 for which I am training a CART decision tree. My dataset has about 30 variables and the outcome has 3 classes. I am using CART for interpretability purposes, ...
Eric Yamga's user avatar
0 votes
1 answer
29 views

How to proceed when training data change frequently (in production)?

I'm working with a Recommendation System that would take as parameters a bunch of "tweets" a user see during his navegation on a mobile app. Every tweet has a property, like a category (...
Antonio Carlos's user avatar
1 vote
0 answers
21 views

What to do if your adversarial validation show different distributions for an NLP problem?

I was trying to figure out if the test set from a competition is similar to the train set. This was done in a NLP competition, in which I had two columns, tweet and type, and I needed to predict the ...
dsbr__0's user avatar
  • 191
0 votes
1 answer
986 views

Training loss decreasing while Validation loss is not decreasing

I am wondering why validation loss of this regression problem is not decreasing while I have implemented several methods such as making the model simpler, adding early stopping, various learning rates,...
ali khorshidian's user avatar
1 vote
1 answer
164 views

Model Performance on external validation Set really low?

I am using the LGBM model for binary classification. My train and test accuracies are 87% & 82% respectively with cross-validation of 89%. ROC-AUC score of 81%. But when evaluating model ...
As13's user avatar
  • 77
0 votes
1 answer
164 views

What does it mean if the validation accuracy is equal to the testing accuracy?

I am training a CNN model for my specific problem. I have divided the dataset into 70% training set, 20% validation set, and 10% test set. The validation accuracy achieved was 95% and the test ...
AAA's user avatar
  • 145
1 vote
0 answers
31 views

How can I calculate de AUC PR of my classifiers in a multiclass scenario?

I'm developing image classifiers in a context with 25k images and 50 classes. The dataset is imbalanced. Some papers recommend AUC PR for comparing the performance of my classifiers in this setting. ...
Zaratruta's user avatar
  • 139
0 votes
1 answer
143 views

Training Loss or Validation Loss for Hyperparameter Optimisation

When performing HO, should I be looking to train each model (each with different hyperparameter values, e.g. with RandomSearch picking those values) on the training data, and then the best one is ...
Socorro's user avatar
  • 111
1 vote
0 answers
23 views

Is Callback / early stopping and validation set is not mandatory

I just noticed that in mostly github repositry of research papers they didnt implemented early stopping criteria and they didnt use validation set but whats the reason behind this?
user12's user avatar
  • 171
0 votes
1 answer
23 views

How to print the corresponding c of the lowest classification error on the validation data

I'm currently measuring the overall classification error for an SVM classifier and I'm varying the regularization value C. In the following code, how can I print in ...
104205038's user avatar
1 vote
1 answer
1k views

Why am I getting different prediction result after every run?

I have a simple lstm model ...
Stupid_Intern's user avatar
1 vote
3 answers
185 views

How to do modelling for pairs of non i.i.d. data?

I have a dataset in which i have the labels for candidates on whether they would be hired,interviewed_and_failed,not_interviewed_at_all. The task is to predict for new jobs/new candidates what these ...
Gary Ong's user avatar
2 votes
3 answers
532 views

What is exactly the difference between Validation data and Testing data

I asked this question on stack overflow and was told that this is a better place for it. I am confused with the terms validation and testing, is validating the model same as testing it? is it possible ...
besa's user avatar
  • 33
1 vote
2 answers
137 views

Updating a train/val/test set

It is considered best practice to split your data into a train and test set at the start of a data science / machine learnign project (and then your train set further into a validation set for ...
Aesir's user avatar
  • 458
0 votes
0 answers
33 views

Validation accuracy have an almost good accuracy but loss function is high too

Based on my project, I find a little problem there with the statement like this. I want to make model with neural networks for text dataset. Then I use Pad Sequence for my text and Array for the ...
grace's user avatar
  • 13
2 votes
2 answers
361 views

Dataset and why use evaluate()?

I am starting in Machine Learning, and I have doubts about some concepts. I've read we need to split our dataset into training, validation and test sets. I'll ask four questions related to them. 1 - ...
Murilo's user avatar
  • 125
0 votes
1 answer
697 views

Logarithmic scale for a learning curve [closed]

I'm plotting the learning curve with Python with the following code: ...
Bambeil's user avatar
  • 111
1 vote
1 answer
164 views

Difference between model score on test part and Kaggle public score

I tested my CatBoostModel model on part of data and get 0.92 score, but Kaggle public score was 0.9. I found new hyperparameters via randomsearch, new model score was 0.925, but on Kaggle score fell ...
Dmitry  Sokolov's user avatar
1 vote
0 answers
19 views

Different data dimension when indexing with R

I am facing on something that I cannot explain, so may be someone could explain me. I have a dataset which contain 102011 data. I want to take 70% for the train and 30% remaining for the validation. I ...
user979974's user avatar
1 vote
3 answers
441 views

Spliting Training Test and Validation for Image Dataset

I have 600 images in the training folder, 200 images in the validation folder, and 200 images in the test folder. Suppose if I fit the training data generator and validation data generator for some ...
User's user avatar
  • 36
3 votes
1 answer
2k views

Does validation data has any effect on training or it acts solely without affecting the training?

When using Keras library of Python, we use validation data with training data while training our model. In every epoch, we get a validation accuracy. Does this validation accuracy have any effect on ...
Rawnak Yazdani's user avatar
1 vote
1 answer
40 views

Measure performance of classification model for training on different snapshots

I am trying to do binary classification on some chronological data. Let's assume we have weekly data from the first week of 2017 through the last week of 2020. Now we have found out that 26 weeks of ...
Ricky's user avatar
  • 189
1 vote
1 answer
78 views

Using Z-test score to evaluate model performance

I think I know the answer to this question but I am looking for a sanity check here: Is it appropriate to use z-test scores in order to evaluate the performance of my model? I have a binary model that ...
I_Play_With_Data's user avatar
1 vote
1 answer
279 views

Does it make sense to repeat calculating AUC in logistic regression?

I have a question regarding logistic regression models and testing its skill. I am not quite sure if I understand correctly how the ROC Curve is established. When calculating the ROC curve, is a train ...
DataVader's user avatar
2 votes
0 answers
229 views

ValueError: Input contains NaN, infinity or a value too large for dtype('float32'). Any advice? [closed]

The error below presented itself when attempting to assemble a PCA. My code: ...
Macgregorfb's user avatar
0 votes
1 answer
562 views

Keras: Very high loss for Autoencoder

I am trying to implement an autoencoder for prediction of multiple labels using Keras. This is a snippet: ...
Animeartist's user avatar
0 votes
0 answers
69 views

How many epochs should all the data be trained on after training with validation finds when validation and training diverge?

One uses and train/test split to use their training data to get an idea of how many epochs to train with. If the validation accuracy starts going down while the training accuracy is still going up, ...
user14094230's user avatar
1 vote
2 answers
788 views

dataset split for image classification

I am trying to do image classification for 14 categories (around 1000 images for each cat). And i initially created two folders for training and validation. In this case, do I still need to set a ...
Hello-experts's user avatar
0 votes
0 answers
92 views

Validation accuracy does not increase for binary classification using GNN

I am trying to perform graph classification with GINConv model of GNN. I have tried everything from varying dropouts to weight decay (for L2 regularization), learning rate (1e-6 to 1e-3), batch ...
user3598542's user avatar