67

Really great question, and one that I find that most people don't really understand on an intuitive level. AUC is in fact often preferred over accuracy for binary classification for a number of different reasons. First though, let's talk about exactly what AUC is. Honestly, for being one of the most widely used efficacy metrics, it's surprisingly obtuse to ...


20

In the context of Kaggle, it means LeaderBoard (emphasis mine).


20

There are a couple of nuances here. Complexity question very important - ocams razor CV - is this trully the case 84%/83% (test it for train+test with CV) Given this, personal opinion: Second one. Better to catch general patterns. You already know that first model failed on that because of the train and test difference. 1% says nothing.


19

AUC and accuracy are fairly different things. AUC applies to binary classifiers that have some notion of a decision threshold internally. For example logistic regression returns positive/negative depending on whether the logistic function is greater/smaller than a threshold, usually 0.5 by default. When you choose your threshold, you have a classifier. You ...


17

Definitions Accuracy: The amount of correct classifications / the total amount of classifications. The train accuracy: The accuracy of a model on examples it was constructed on. The test accuracy is the accuracy of a model on examples it hasn't seen. Confusion matrix: A tabulation of the predicted class (usually vertically) against the actual class (thus ...


15

Interesting question. I personally haven't seen that for products going into production, but understand the logic. Theoretically, the more data your deployed model has seen, the better is should generalise. So if you trained the model on the full set of data you have available, it should generalise better than a model which only saw for example train/val ...


14

Once you have obtained optimal hyperparamters for your model, after training and cross validating etc., in theory it is ok to train the model on the entire dataset to deploy to production. This will, in theory, generalise better. HOWEVER, you can no longer make statistical / performance claims on test data since you no longer have a test dataset. If you ...


13

As long as your validation accuracy increases, you should keep training. I would stop when the test accuracy starts decreasing (this is known as early stopping). The general advise is always to keep the model that performs the best in your validation set. Although it is right that your model overfits a little since epoch 280, it is not necessarily a bad ...


12

Rational business people don't pay for accuracy, they pay to either save money on a profitable process (thereby making it more profitable), or by creating new money (creating new profitable processes). So any project that is undertaken has to be couched in terms that reflect this. The first step is always understanding which of the two processes ...


11

It depends mostly on the problem context. If predictive performance is all you care about, and you believe the test set to be representative of future unseen data, then the first model is better. (This might be the case for, say, health predictions.) There are a number of things that would change this decision. Interpretability / explainability. This is ...


10

Gather competitive counterparts. Try and determine a state-of-the-art and see how your models compare with that. It also heavily depends on how long your team has been working on it. Science-driven models are not created statically, they develop dynamically because a good scientist will always try to find ways to improve it. Upper management personnel ...


8

Which two accuracies I compare to see if the model is overfitting or not? You should compare the training and test accuracies to identify over-fitting. A training accuracy that is subjectively far higher than test accuracy indicates over-fitting. Here, "accuracy" is used in a broad sense, it can be replaced with F1, AUC, error (increase becomes decrease, ...


6

A point that needs to be emphasized about statistical machine learning is that there are no guarantees. When you estimate performance using a held-out set, that is just an estimate. Estimates can be wrong. This takes some getting used to, but it's something you're going to have to get comfortable with. When you say "What if the performance actually ...


6

I think this is because your targets y are continuous instead of binary. Therefore, either ignore the accuracy report, or binarize your targets if applicable. I assumed you are using Keras. When you use metrics=['accuracy'], this is what happens under the hood: if metric in ('accuracy', 'acc'): metric_fn = metrics_module.binary_accuracy where def ...


5

I like this question because it gets at the politics that exist in every organization. In my view and to a significant degree, expectations about model performance are a function of the org culture and degree to which an organization is "technically literate." One way to make clear what I mean is to consider the differences between the 4 big "data science" ...


5

I expected more from random trees: With random forests, typically for N features, sqrt(N) features are used for each decision tree construction. Since in your case N=20, you could try setting max_depth (the number of sub-features to construct each decision tree) to 5. Instead of decision trees, linear models have been proposed and evaluated as base ...


5

The term overfitting means the model is learning relationships between attributes that only exist in this specific dataset and do not generalize to new, unseen data. Just by looking at the model accuracy on the data that was used to train the model, you won't be able to detect if your model is or isn't overfitting. To see if you are overfitting, split your ...


5

Yes and No! depending on what do you mean by minimization. When you say minimizing $f$ and $g$ according to something, you are actually looking for a point which minimizes both. It does not mean that this point necessarily finds the minimum of $f$ or $g$. So yes in this sense. But in case you mean a point in which both of them are in their minimum, this ...


5

There may be a few reason this is happening. First of all, check your code. 100% accuracy seems unlikely in any setting. How many testing data points do you have? How many training data points did you train your model on? You may have made a coding mistake and compared two same list. Did you use different test set for testing? The high accuracy may be due ...


5

You haven't normalized your image dataset such as setting the pixel values between 0-1 which could help classifier converge faster. Please do it by doing the operation below. train_images = train_images / 255.0 test_images = test_images / 255.0 It seems you are using 50% of data for training as well as testing. try to use the data in the ratio of 7:3 for ...


5

For visualizing paired 1D data, i.e. true vs predicted counts, you may use something like ggpaired. You may also visualize the distribution of differences, where each sample is the difference between the true count and its predicted value; excluding the zero differences could better accentuate any deviation. For a statistical test, you may use Wilcoxon ...


5

You can! The trick is that you actually know two other critical variables: the number of positive and negative examples (P and N). You can then use them to algebraically solve for the confusion matrix: $ recall=\frac{TP}{TP+FN}=1-\frac{FN}{P}\Rightarrow $ $ FN = P(1-recall) $ $ recall=\frac{TP}{TP+FN}\Rightarrow (recall)(TP+FN)=TP\Rightarrow TP(1-recall)=...


5

The first has an accuracy of 100% on training set and 84% on test set. Clearly over-fitted. Maybe not. It's true that 100% training accuracy is usually a strong indicator of overfitting, but it's also true that an overfit model should perform worse on the test set than a model that isn't overfit. So if you're seeing these numbers, something unusual is going ...


4

Yes, you can do all this using the Caret (http://caret.r-forge.r-project.org/training.html) package in R. For example, fitControl <- trainControl(## 10-fold CV method = "repeatedcv", number = 10, ## repeated ten times repeats = 10) gbmFit1 <- ...


4

I'd like to refer to how you should choose a performance measure. Before that I'll refer to the specific question of accuracy and AUC. As answered before, on imbalanced dataset using the majority run as a classifier will lead to high accuracy what will make it a misleading measure. AUC aggregate over confidence threshold, for good and bad. For good, you get ...


4

I do not know a standard answer to this, but I thought about it some times ago and I have some ideas to share. When you have one confusion matrix, you have more or less a picture of how you classification model confuse (mis-classify) classes. When you repeat classification tests you will end up having multiple confusion matrices. The question is how to get ...


4

There are a few ways to achieve your "master confusion matrix". Sum all the confusion matrices together: Like you suggested, summing this results in a confusion matrix. The problem with this is you can not interpret totals. Average the entries. This method is the same as number one, but you divide each entry by the number of trials (~400 in your case). ...


4

Cross validation can be used in parameter tuning or model selection, but it does not evaluate the performance of a model. When developing a model, you divide your data between train, validation and testing. In the best case scenario, testing is only used once at the end to score the model. You should definitely keep the 2016 data. If you give all your ...


4

The Gini Coefficient can also be expressed in terms of the area under the ROC curve (AUC): G = 2*AUC -1 link. The ROC curve, on the other hand, is influenced by class imbalance through the false positive rate FP/(FP+TN). If the number of negatives is a lot larger, this could be a potential issue. In short, the Gini Coefficient has similar pros and cons as ...


Only top voted, non community-wiki answers of a minimum length are eligible