84 votes
Accepted

Advantages of AUC vs standard accuracy

Really great question, and one that I find that most people don't really understand on an intuitive level. AUC is in fact often preferred over accuracy for binary ...
  • 4,189
34 votes
Accepted

What is a LB score in machine learning?

In the context of Kaggle, it means LeaderBoard (emphasis mine).
  • 10.4k
27 votes

Advantages of AUC vs standard accuracy

AUC and accuracy are fairly different things. AUC applies to binary classifiers that have some notion of a decision threshold internally. For example logistic regression returns positive/negative ...
  • 6,425
25 votes
Accepted

Train Accuracy vs Test Accuracy vs Confusion matrix

Definitions Accuracy: The amount of correct classifications / the total amount of classifications. The train accuracy: The accuracy of a model on examples it was constructed on. The test accuracy ...
  • 1,344
24 votes

Is it always better to use the whole dataset to train the final model?

A point that needs to be emphasized about statistical machine learning is that there are no guarantees. When you estimate performance using a held-out set, that is just an estimate. Estimates can be ...
  • 3,037
23 votes

macro average and weighted average meaning in classification_report

Your data set is unbalanced since 28432 out of 28481 examples belong to class 0 (that is 99.8%). Therefore, your predictor almost always predicts any given sample as belonging to class 0 and thereby ...
  • 5,055
22 votes

Is it always better to use the whole dataset to train the final model?

I personally haven't seen that for products going into production, but understand the logic. Theoretically, the more data your deployed model has seen, the better is should generalise. So if you ...
  • 14.1k
21 votes

What would I prefer - an over-fitted model or a less accurate model?

There are a couple of nuances here. Complexity question very important - ocams razor CV - is this trully the case 84%/83% (test it for train+test with CV) Given this, personal opinion: Second one. ...
  • 5,369
20 votes
Accepted

In which epoch should i stop the training to avoid overfitting

As long as your validation accuracy increases, you should keep training. I would stop when the test accuracy starts decreasing (this is known as early stopping). The general advise is always to keep ...
  • 5,714
19 votes

Is it always better to use the whole dataset to train the final model?

Once you have obtained optimal hyperparamters for your model, after training and cross validating etc., in theory it is ok to train the model on the entire dataset to deploy to production. This will, ...
  • 1,514
19 votes
Accepted

What are the disadvantages of accuracy?

A common complaint about accuracy is that it fails when the classes are imbalanced. For instance, if you get an accuracy of $98\%$, that sounds like a high $\text{A}$ in school, so you might be pretty ...
  • 3,278
15 votes
Accepted

How to know if a model is overfitting or underfitting by looking at graph

Overfitting is a scenario where your model performs well on training data but performs poorly on data not seen during training. This basically means that your model has memorized the training data ...
  • 2,434
15 votes

What would I prefer - an over-fitted model or a less accurate model?

It depends mostly on the problem context. If predictive performance is all you care about, and you believe the test set to be representative of future unseen data, then the first model is better. (...
  • 10.1k
12 votes

How do you manage expectations at work?

Rational business people don't pay for accuracy, they pay to either save money on a profitable process (thereby making it more profitable), or by creating new money (creating new profitable ...
11 votes

My data is highly overlapping, but when I apply logistic regression, it is giving an impressive accuracy of 79%. Why?

Decision Tree, KNN, & Random Forest (Methods that are suitable for overlapping data) This statement is false. All those methods are good when the decision surface (separating surface) has a ...
  • 4,448
10 votes

How do you manage expectations at work?

Gather competitive counterparts. Try and determine a state-of-the-art and see how your models compare with that. It also heavily depends on how long your team has been working on it. Science-driven ...
10 votes
Accepted

Balanced Accuracy vs. F1 Score

One major difference is that the F1-score does not care at all about how many negative examples you classified or how many negative examples are in the dataset at all; instead, the balanced accuracy ...
  • 773
9 votes
Accepted

Validation vs. test vs. training accuracy. Which one should I compare for claiming overfit?

Which two accuracies I compare to see if the model is overfitting or not? You should compare the training and test accuracies to identify over-fitting. A training accuracy that is subjectively far ...
  • 8,667
9 votes

What would I prefer - an over-fitted model or a less accurate model?

The first has an accuracy of 100% on training set and 84% on test set. Clearly over-fitted. Maybe not. It's true that 100% training accuracy is usually a strong indicator of overfitting, but it's ...
  • 191
8 votes

99% validation accuracy but 0% prediction results (UNET Architecture)

There is no "mismatch" of accuracy. Your problem is that you have an image segmentation problem where 99% of the pixels should be zero. So getting 99% accuracy is trivially easy. A model that predicts ...
  • 27.6k
8 votes
Accepted

I got 100% accuracy on my test set,is there something wrong?

There may be a few reason this is happening. First of all, check your code. 100% accuracy seems unlikely in any setting. How many testing data points do you have? How many training data points did ...
  • 146
8 votes

Accuracy for Kmeans clustering

Accuracy is a measure of comparing the true label to the predicted label. K-Means is an unsupervised clustering algorithm where a predicted label does not exist. So, accuracy can not be directly ...
  • 284
8 votes

What are the disadvantages of accuracy?

In general, the main disadvantage of accuracy is that it masks the issue of class imbalance. For example if the data contains only 10% of positive instances, a majority baseline classifier which ...
  • 22.7k
7 votes
Accepted

0.1 accuracy on MNIST fashion dataset following official Tensorflow/Keras tutorial

You haven't normalized your image dataset such as setting the pixel values between 0-1 which could help classifier converge faster. Please do it by doing the operation below. ...
7 votes
Accepted

Inverse Relationship Between Precision and Recall

If we decrease the false negative (select more positives), recall always increases, but precision may increase or decrease. Generally, for models better than random, precision and recall have an ...
  • 8,667
6 votes
Accepted

Is Gini coefficient a good metric for measuring predictive model performance on highly imbalanced data

The Gini Coefficient can also be expressed in terms of the area under the ROC curve (AUC): G = 2*AUC -1 link. The ROC curve, on the other hand, is influenced by ...
  • 6,035
6 votes
Accepted

How to determine if my GBM model is overfitting?

The term overfitting means the model is learning relationships between attributes that only exist in this specific dataset and do not generalize to new, unseen data. Just by looking at the model ...
6 votes

Validation vs. test vs. training accuracy. Which one should I compare for claiming overfit?

Cross validation splits your data into K folds. Each fold contains a set of training data and test data. You are correct that you get K different error rates that you then take the mean of. These ...
  • 337
6 votes

loss/val_loss are decreasing but accuracies are the same in LSTM!

I think this is because your targets y are continuous instead of binary. Therefore, either ignore the accuracy report, or ...
  • 8,667

Only top scored, non community-wiki answers of a minimum length are eligible