35
votes
Accepted
What is a LB score in machine learning?
In the context of Kaggle, it means LeaderBoard (emphasis mine).
- 10.5k
28
votes
Is it always better to use the whole dataset to train the final model?
A point that needs to be emphasized about statistical machine learning is that there are no guarantees. When you estimate performance using a held-out set, that is just an estimate. Estimates can be ...
- 3,232
25
votes
Accepted
Train Accuracy vs Test Accuracy vs Confusion matrix
Definitions
Accuracy: The amount of correct classifications / the total amount
of classifications.
The train accuracy: The accuracy of a model on examples it was constructed on.
The test accuracy ...
- 1,344
24
votes
macro average and weighted average meaning in classification_report
Your data set is unbalanced since 28432 out of 28481 examples belong to class 0 (that is 99.8%). Therefore, your predictor almost always predicts any given sample as belonging to class 0 and thereby ...
- 5,230
22
votes
Is it always better to use the whole dataset to train the final model?
I personally haven't seen that for products going into production, but understand the logic.
Theoretically, the more data your deployed model has seen, the better is should generalise. So if you ...
- 14.4k
21
votes
What would I prefer - an over-fitted model or a less accurate model?
There are a couple of nuances here.
Complexity question very important - ocams razor
CV - is this trully the case 84%/83% (test it for train+test with CV)
Given this, personal opinion: Second one.
...
- 5,519
20
votes
Accepted
In which epoch should i stop the training to avoid overfitting
As long as your validation accuracy increases, you should keep training. I would stop when the test accuracy starts decreasing (this is known as early stopping). The general advise is always to keep ...
- 5,921
19
votes
Is it always better to use the whole dataset to train the final model?
Once you have obtained optimal hyperparamters for your model, after training and cross validating etc., in theory it is ok to train the model on the entire dataset to deploy to production. This will, ...
- 1,564
19
votes
Accepted
What are the disadvantages of accuracy?
A common complaint about accuracy is that it fails when the classes are imbalanced. For instance, if you get an accuracy of $98\%$, that sounds like a high $\text{A}$ in school, so you might be pretty ...
- 3,744
15
votes
Accepted
How to know if a model is overfitting or underfitting by looking at graph
Overfitting is a scenario where your model performs well on training data but performs poorly on data not seen during training. This basically means that your model has memorized the training data ...
- 2,564
15
votes
What would I prefer - an over-fitted model or a less accurate model?
It depends mostly on the problem context. If predictive performance is all you care about, and you believe the test set to be representative of future unseen data, then the first model is better. (...
- 10.8k
12
votes
How do you manage expectations at work?
Rational business people don't pay for accuracy, they pay to either
save money on a profitable process (thereby making it more profitable), or by
creating new money (creating new profitable ...
11
votes
Accepted
Balanced Accuracy vs. F1 Score
One major difference is that the F1-score does not care at all about how many negative examples you classified or how many negative examples are in the dataset at all; instead, the balanced accuracy ...
- 803
11
votes
My data is highly overlapping, but when I apply logistic regression, it is giving an impressive accuracy of 79%. Why?
Decision Tree, KNN, & Random Forest (Methods that are suitable for overlapping data)
This statement is false. All those methods are good when the decision surface (separating surface) has a ...
- 4,593
10
votes
How do you manage expectations at work?
Gather competitive counterparts. Try and determine a state-of-the-art and see how your models compare with that. It also heavily depends on how long your team has been working on it. Science-driven ...
10
votes
Accepted
Validation vs. test vs. training accuracy. Which one should I compare for claiming overfit?
Which two accuracies I compare to see if the model is overfitting or not?
You should compare the training and test accuracies to identify over-fitting. A training accuracy that is subjectively far ...
- 8,967
9
votes
What would I prefer - an over-fitted model or a less accurate model?
The first has an accuracy of 100% on training set and 84% on test set. Clearly over-fitted.
Maybe not. It's true that 100% training accuracy is usually a strong indicator of overfitting, but it's ...
- 191
8
votes
99% validation accuracy but 0% prediction results (UNET Architecture)
There is no "mismatch" of accuracy. Your problem is that you have an image segmentation problem where 99% of the pixels should be zero. So getting 99% accuracy is trivially easy. A model that predicts ...
- 28.1k
8
votes
Accepted
I got 100% accuracy on my test set,is there something wrong?
There may be a few reason this is happening.
First of all, check your code. 100% accuracy seems unlikely in any setting. How many testing data points do you have? How many training data points did ...
- 146
8
votes
Accuracy for Kmeans clustering
Accuracy is a measure of comparing the true label to the predicted label. K-Means is an unsupervised clustering algorithm where a predicted label does not exist. So, accuracy can not be directly ...
- 294
8
votes
What are the disadvantages of accuracy?
In general, the main disadvantage of accuracy is that it masks the issue of class imbalance. For example if the data contains only 10% of positive instances, a majority baseline classifier which ...
- 24.5k
7
votes
Accepted
0.1 accuracy on MNIST fashion dataset following official Tensorflow/Keras tutorial
You haven't normalized your image dataset such as setting the pixel values between 0-1 which could help classifier converge faster.
Please do it by doing the operation below.
...
7
votes
Accepted
Inverse Relationship Between Precision and Recall
If we decrease the false negative (select more positives), recall always increases, but precision may increase or decrease. Generally, for models better than random, precision and recall have an ...
- 8,967
6
votes
Accepted
Is Gini coefficient a good metric for measuring predictive model performance on highly imbalanced data
The Gini Coefficient can also be expressed in terms of the area under the ROC curve (AUC): G = 2*AUC -1 link. The ROC curve, on the other hand, is influenced by ...
oW_♦
- 6,134
6
votes
Accepted
How to determine if my GBM model is overfitting?
The term overfitting means the model is learning relationships between attributes that only exist in this specific dataset and do not generalize to new, unseen data.
Just by looking at the model ...
- 371
6
votes
Accepted
human level performance on ImageNet, top-1 or top-5?
It comes from this paper: https://arxiv.org/abs/1409.0575
O. Russakovsky "ImageNet Large Scale Visual Recognition Challenge" 2014
- 1,229
6
votes
Validation vs. test vs. training accuracy. Which one should I compare for claiming overfit?
Cross validation splits your data into K folds. Each fold contains a set of training data and test data. You are correct that you get K different error rates that you then take the mean of. These ...
- 337
6
votes
loss/val_loss are decreasing but accuracies are the same in LSTM!
I think this is because your targets y are continuous instead of binary. Therefore, either ignore the accuracy report, or ...
- 8,967
6
votes
Is it better to have higher train accuracy with lower test accuracy or higher test accuracy with lower train accuracy?
Test accuracy better reflects generalization error, so you want the one with higher test accuracy. In your first setup, the higher train accuracy indicates overfitting, as it's significantly higher ...
- 6,515
6
votes
My data is highly overlapping, but when I apply logistic regression, it is giving an impressive accuracy of 79%. Why?
Your data is multidimensional, it is possible that any two dimensional projection overlaps while still existing an hyperplane on the original dimensionality that separates the two classes well
Say for ...
- 181
Only top scored, non community-wiki answers of a minimum length are eligible