Skip to main content
5 votes
Accepted

Validation and training loss of a model are not stable

It seems like you're over fitting. There are tones of articles and blogs on how to avoid over fitting, but I mention some of them here anyway: Reduce your learning rate to a very small number like 0....
Amirhossein Rezaei's user avatar
5 votes

How to build a model when we have three separate train, validation, and test sets?

The validation set would be used for the same job as the split in cross-validation, except that it's done only once: For each different assignment for the variables, train on the training set, then ...
Erwan's user avatar
  • 25.4k
4 votes
Accepted

Dataset and why use evaluate()?

First to be clear we're talking specifically about supervised learning here: there's a training stage during which the model is provided with labelled instances (features and class). Simple analogy: ...
Erwan's user avatar
  • 25.4k
4 votes
Accepted

What is exactly the difference between Validation data and Testing data

Usually you first split your dataset into train/test set, and then if your model training process requires a validation set, you can further split your train-set into the final train-set and the ...
Raymond Kwok's user avatar
3 votes
Accepted

Does validation data has any effect on training or it acts solely without affecting the training?

To make it short - no. When you train a model (e.g a Neural Network) you parse some data X into the model, the model predicts something, $\hat{y}$, you look at the ...
CutePoison's user avatar
2 votes
Accepted

Using the whole dataset for testing (not validation) in case of small datasets

Considering that you have not used a cross-validation strategy, you could try to use the LOOCV (Leave One Out Cross Validation) strategy, so you have several splits (as many as samples considered in ...
German C M's user avatar
  • 2,696
2 votes
Accepted

Why do machine learning engineers insist on training with more data than validation set?

The reasoning will be: "The more data for training the better". Then you have to keep in mind that the validation/hold-out set has to resemble how it should work on production/testing. The ...
Carlos Mougan's user avatar
2 votes

Does it make sense to repeat calculating AUC in logistic regression?

The calculation of ROC curve and the AUC based off of that curve is simply a comparison of the predictions from your model (logistic regression) and the actual values on some set of data. This can ...
engelAnalytics's user avatar
2 votes

Dataset and why use evaluate()?

Training dataset is used to train a model to learn information from data to solve a problem(Prediction). Validation dataset is used for checking whether the trained model is good enough to solve the ...
Udaya Unnikrishnan's user avatar
2 votes
Accepted

Logarithmic scale for a learning curve

This is more of a programming question than a data science question and would therefore be better suited to the stackoverflow stackexchange. To change the y-axis from a linear scale to a logarithmic ...
Oxbowerce's user avatar
  • 7,507
2 votes
Accepted

Difference between model score on test part and Kaggle public score

In general, you should expect to get lower scores on test sets than validation sets, since you took advantage of validation data to tune your model. But for a correctly trained model, the difference ...
SimplyFarzad's user avatar
2 votes

Model Performance on external validation Set really low?

First, an AUC less than 50% is terrible: it means that you get better performance by switching the positive and negative labels! So the model is doing worse than nothing on this data. In general there ...
Erwan's user avatar
  • 25.4k
2 votes
Accepted

Why am I getting different prediction result after every run?

Do you specify the random seed anywhere in your code? If you don't, that might be the explanation why your RMSE value differs on each run for your train/test datasets. You could use the ...
Lars's user avatar
  • 120
2 votes

Is it a problem to use the test dataset for the hyperparameter tuning, when I want to compare 2 classification algorithms on the 10 different dataset?

Seems like there is something flawed in the procedure here. If you use the test data set for tuning, then what do you use for testing performance? In general, the models should not get any information ...
noNameTed's user avatar
2 votes

Updating a train/val/test set

There are two really different scenarios: The training and test data are obtained from the same dataset If the data has been randomly split between training and test set, this is extremely unlikely to ...
Erwan's user avatar
  • 25.4k
2 votes
Accepted

Optimal Number of Epochs for Training Transformer Network on Time series data? Early Stopping and Model Selection Strategies

Unfortunately, time series needs some data preprocessing in order to be efficient. Furthermore, some NN are more noise-sensitive than others. Others are more adapted to medium-term prediction or short-...
Nicolas Martin's user avatar
1 vote

how to fix my increasing validation loss and decreasing training loss?

I agree with noe that the model is overfitting. Try adjusting the hyperparameters. Reduce the learning rate, try more epochs. You can also try using a dropout layer. Is it possible that your dataset ...
Nemo_the_scientist's user avatar
1 vote

Testing RANSAC regression model

There are different aspects to consider here: 1. Robustness One of the reasons to use RANSAC is its robustness towards outliers. That means that some outliers more or less in the training set will ...
Broele's user avatar
  • 1,362
1 vote

In cross validation, should the test dataset not be fixed

What you are trying to do is called Nested cross-validation. Nested cross-validation is an approach to model hyperparameter optimization and model selection that attempts to overcome the problem of ...
Amirhossein Rezaei's user avatar
1 vote

Training loss decreasing while Validation loss is not decreasing

Looks like the model overfits, very rapidly (in just a few epoches). I would start with combining all approaches you mentioned: making the model simpler, adding early stopping, various learning rates, ...
lpounng's user avatar
  • 1,018
1 vote

How to do modelling for pairs of non i.i.d. data?

In order to have an appropriate model validation, you should always think of how the inference/prediction phase will work once your model is in production. In this case, if I did not misunderstand, ...
Multivac's user avatar
  • 2,969
1 vote
Accepted

How to do modelling for pairs of non i.i.d. data?

I would use leave one out, where the partition is on a per-job basis. Completely isolate a particular job for testing Divide your remaining data into training and validation data as you see fit. ...
Warlax56's user avatar
  • 430
1 vote

What is exactly the difference between Validation data and Testing data

This question is evidence that the scientific method has sort of gotten lost in the way the ML world communicates about models, and it causes students to get confused when they enter industry jobs. ...
Paul Siegel's user avatar
1 vote

What is exactly the difference between Validation data and Testing data

You can use the testing data to perform hyperparameters optimization to see which hyperparameters of your model pipeline work the best. The validation data is then only used once to see how the whole ...
Oxbowerce's user avatar
  • 7,507
1 vote
Accepted

Measure performance of classification model for training on different snapshots

The average performance is a good summary of the performance, but you should also mention the variations across time. For example you could calculate the standard deviation across the snapshots. I ...
Erwan's user avatar
  • 25.4k
1 vote

Spliting Training Test and Validation for Image Dataset

It's bad practice to train a model and not have an independent way to evaluate its suitability or performance relative to a metric. It's tempting to think that adding more data produces a better model ...
fswings's user avatar
  • 378
1 vote

dataset split for image classification

If you have already split your training and validation sets into separate directories then there is no need to technically do the splitting in your code. However, the problem with a pre-defined ...
VRaina's user avatar
  • 66
1 vote

How to compare a machine learning model and a rule based model

For the rule base model I would take the exact same approach as for ML model, split the data into train and test set since at the end you want to check whether or not your rule base and ml models ...
Multivac's user avatar
  • 2,969
1 vote
Accepted

Does overfitting depend only on validation loss or both training and validation loss?

In my opinion, only case 3 should be considered overfitting. As @stans has mentioned, there is not a very rigorous definition of overfitting so other people might think differently. I wouldn't say the ...
David Masip's user avatar
  • 6,061
1 vote
Accepted

Is it right to maintain the train distribution in test set for unbalanced data?

If the training set was unbalanced the chances are the model will be biased. Not really. Depending on the loss function you use. Also, note that for data to be unbalanced at least it has to be in a ...
Carlos Mougan's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible