82

Really great question, and one that I find that most people don't really understand on an intuitive level. AUC is in fact often preferred over accuracy for binary classification for a number of different reasons. First though, let's talk about exactly what AUC is. Honestly, for being one of the most widely used efficacy metrics, it's surprisingly obtuse to ...


30

In the context of Kaggle, it means LeaderBoard (emphasis mine).


26

AUC and accuracy are fairly different things. AUC applies to binary classifiers that have some notion of a decision threshold internally. For example logistic regression returns positive/negative depending on whether the logistic function is greater/smaller than a threshold, usually 0.5 by default. When you choose your threshold, you have a classifier. You ...


25

Definitions Accuracy: The amount of correct classifications / the total amount of classifications. The train accuracy: The accuracy of a model on examples it was constructed on. The test accuracy is the accuracy of a model on examples it hasn't seen. Confusion matrix: A tabulation of the predicted class (usually vertically) against the actual class (thus ...


21

There are a couple of nuances here. Complexity question very important - ocams razor CV - is this trully the case 84%/83% (test it for train+test with CV) Given this, personal opinion: Second one. Better to catch general patterns. You already know that first model failed on that because of the train and test difference. 1% says nothing.


19

As long as your validation accuracy increases, you should keep training. I would stop when the test accuracy starts decreasing (this is known as early stopping). The general advise is always to keep the model that performs the best in your validation set. Although it is right that your model overfits a little since epoch 280, it is not necessarily a bad ...


19

I personally haven't seen that for products going into production, but understand the logic. Theoretically, the more data your deployed model has seen, the better is should generalise. So if you trained the model on the full set of data you have available, it should generalise better than a model which only saw for example train/val sets (e.g. ~ 90%) from ...


17

Once you have obtained optimal hyperparamters for your model, after training and cross validating etc., in theory it is ok to train the model on the entire dataset to deploy to production. This will, in theory, generalise better. HOWEVER, you can no longer make statistical / performance claims on test data since you no longer have a test dataset. If you ...


15

A point that needs to be emphasized about statistical machine learning is that there are no guarantees. When you estimate performance using a held-out set, that is just an estimate. Estimates can be wrong. This takes some getting used to, but it's something you're going to have to get comfortable with. When you say "What if the performance actually ...


14

Your data set is unbalanced since 28432 out of 28481 examples belong to class 0 (that is 99.8%). Therefore, your predictor almost always predicts any given sample as belonging to class 0 and thereby achieves very high scores like precision and recall for class 0 and very low scores for class 1. In the case of weighted average the performance metrics are ...


14

It depends mostly on the problem context. If predictive performance is all you care about, and you believe the test set to be representative of future unseen data, then the first model is better. (This might be the case for, say, health predictions.) There are a number of things that would change this decision. Interpretability / explainability. This is ...


12

Rational business people don't pay for accuracy, they pay to either save money on a profitable process (thereby making it more profitable), or by creating new money (creating new profitable processes). So any project that is undertaken has to be couched in terms that reflect this. The first step is always understanding which of the two processes ...


11

Decision Tree, KNN, & Random Forest (Methods that are suitable for overlapping data) This statement is false. All those methods are good when the decision surface (separating surface) has a highly nonlinear form. They act as a non-parametric local approximation - all parameters are not in fact parameters of the decision function but are meta parameters ...


10

Gather competitive counterparts. Try and determine a state-of-the-art and see how your models compare with that. It also heavily depends on how long your team has been working on it. Science-driven models are not created statically, they develop dynamically because a good scientist will always try to find ways to improve it. Upper management personnel ...


10

Overfitting is a scenario where your model performs well on training data but performs poorly on data not seen during training. This basically means that your model has memorized the training data instead of learning the relationships between features and labels. If you are familiar with the bias/variance tradeoff, then you can think of overfitting as a ...


9

Which two accuracies I compare to see if the model is overfitting or not? You should compare the training and test accuracies to identify over-fitting. A training accuracy that is subjectively far higher than test accuracy indicates over-fitting. Here, "accuracy" is used in a broad sense, it can be replaced with F1, AUC, error (increase becomes decrease, ...


8

There is no "mismatch" of accuracy. Your problem is that you have an image segmentation problem where 99% of the pixels should be zero. So getting 99% accuracy is trivially easy. A model that predicts just blank output images would score roughly the same as your network has so far. Your accuracy metric is not meaningful. The low Dice coefficient score gives ...


8

Accuracy is a measure of comparing the true label to the predicted label. K-Means is an unsupervised clustering algorithm where a predicted label does not exist. So, accuracy can not be directly applied to K-Means clustering evaluation. However, there are two examples of metrics that you could use to evaluate your clusters. Within Cluster Sum of Squares ...


8

The first has an accuracy of 100% on training set and 84% on test set. Clearly over-fitted. Maybe not. It's true that 100% training accuracy is usually a strong indicator of overfitting, but it's also true that an overfit model should perform worse on the test set than a model that isn't overfit. So if you're seeing these numbers, something unusual is going ...


7

You haven't normalized your image dataset such as setting the pixel values between 0-1 which could help classifier converge faster. Please do it by doing the operation below. train_images = train_images / 255.0 test_images = test_images / 255.0 It seems you are using 50% of data for training as well as testing. try to use the data in the ratio of 7:3 for ...


7

If we decrease the false negative (select more positives), recall always increases, but precision may increase or decrease. Generally, for models better than random, precision and recall have an inverse relationship (@pythinker's answer), but for models worse than random, they have a direct relationship (@kbrose's example). It is worth noting that we can ...


6

The term overfitting means the model is learning relationships between attributes that only exist in this specific dataset and do not generalize to new, unseen data. Just by looking at the model accuracy on the data that was used to train the model, you won't be able to detect if your model is or isn't overfitting. To see if you are overfitting, split your ...


6

There may be a few reason this is happening. First of all, check your code. 100% accuracy seems unlikely in any setting. How many testing data points do you have? How many training data points did you train your model on? You may have made a coding mistake and compared two same list. Did you use different test set for testing? The high accuracy may be due ...


6

I think this is because your targets y are continuous instead of binary. Therefore, either ignore the accuracy report, or binarize your targets if applicable. I assumed you are using Keras. When you use metrics=['accuracy'], this is what happens under the hood: if metric in ('accuracy', 'acc'): metric_fn = metrics_module.binary_accuracy where def ...


6

Test accuracy better reflects generalization error, so you want the one with higher test accuracy. In your first setup, the higher train accuracy indicates overfitting, as it's significantly higher than train accuracy. This is also kind of why it generalizes less well than the second one.


6

One major difference is that the F1-score does not care at all about how many negative examples you classified or how many negative examples are in the dataset at all; instead, the balanced accuracy metric gives half its weight to how many positives you labeled correctly and how many negatives you labeled correctly. When working on problems with heavily ...


6

Your data is multidimensional, it is possible that any two dimensional projection overlaps while still existing an hyperplane on the original dimensionality that separates the two classes well Say for instance you have 3 data points from 2 labels in 2d that are linearly separable X:(0,-1) O:(1,2) X:(4,3) X O X In the x axis they look ...


5

I like this question because it gets at the politics that exist in every organization. In my view and to a significant degree, expectations about model performance are a function of the org culture and degree to which an organization is "technically literate." One way to make clear what I mean is to consider the differences between the 4 big "data science" ...


5

I expected more from random trees: With random forests, typically for N features, sqrt(N) features are used for each decision tree construction. Since in your case N=20, you could try setting max_depth (the number of sub-features to construct each decision tree) to 5. Instead of decision trees, linear models have been proposed and evaluated as base ...


Only top voted, non community-wiki answers of a minimum length are eligible