21

Definitions Accuracy: The amount of correct classifications / the total amount of classifications. The train accuracy: The accuracy of a model on examples it was constructed on. The test accuracy is the accuracy of a model on examples it hasn't seen. Confusion matrix: A tabulation of the predicted class (usually vertically) against the actual class (thus ...


16

To get a confusion matrix from the test data you should go througt two steps: Make predictions for the test data For example, use model.predict_generator to predict the first 2000 probabilities from the test generator. generator = datagen.flow_from_directory( 'data/test', target_size=(150, 150), batch_size=16, class_mode=...


8

Considering you have two lists y_actual and y_pred ( I assume you made a typo error on x_test and x_pred as in your code), you can pass the two lists to this function to parse them def perf_measure(y_actual, y_pred): TP = 0 FP = 0 TN = 0 FN = 0 for i in range(len(y_pred)): if y_actual[i]==y_pred[i]==1: TP += 1 ...


8

You are just predicting if Play = Yes or Play = No. The confusion matrix would look like this: Predicted +------+------+ | Yes | No | +-------------------+ A | | | | c | Yes | TP | FP | t | | | | u +-------------------+ a | | | | l | No | FN | TN | | ...


8

Multi-class Confusion Matrix is very well established in literature; you could find it easily on your own. Anyhow, Scikit-learn can do it easily like: from sklearn.metrics import confusion_matrix y_true = ['Cat', 'Dog', 'Rabbit', 'Cat', 'Cat', 'Rabbit'] y_pred = ['Dog', 'Dog', 'Rabbit', 'Dog', 'Dog', 'Rabbit'] classes=['Cat', 'Dog', 'Rabbit'] ...


7

F1 will never be zero, but very near to zero for a bad classifier. If TP or TN is zero then there isn't any need to check F1.


6

If we decrease the false negative (select more positives), recall always increases, but precision may increase or decrease. Generally, for models better than random, precision and recall have an inverse relationship (@pythinker's answer), but for models worse than random, they have a direct relationship (@kbrose's example). It is worth noting that we can ...


5

You can apply a technique I described in my masters thesis (page 48ff) and called Confusion Matrix Ordering (CMO): Order the columns/rows in such a way, that most errors are along the diagonal. Split the confusion matrix into multiple blocks such that the single blocks can easily printed / viewed - and such that you can remove some of the blocks because ...


5

Maybe not the answer you would like to hear, but I have to admit that I find wikipedia page on the topic definitively well done... https://en.wikipedia.org/wiki/Precision_and_recall


5

Adding to the answer above, The labeling totally depends on how you define it. You can define 0 as negative or as positive. However, for the sake of understanding and ease of readability, keep it meaningful. The instances that are correctly predicted are given by the diagonal. Here, '1' is True Negative or for the class labelled as 0 and '5' is True ...


5

Yes and No! depending on what do you mean by minimization. When you say minimizing $f$ and $g$ according to something, you are actually looking for a point which minimizes both. It does not mean that this point necessarily finds the minimum of $f$ or $g$. So yes in this sense. But in case you mean a point in which both of them are in their minimum, this ...


5

You can! The trick is that you actually know two other critical variables: the number of positive and negative examples (P and N). You can then use them to algebraically solve for the confusion matrix: $ recall=\frac{TP}{TP+FN}=1-\frac{FN}{P}\Rightarrow $ $ FN = P(1-recall) $ $ recall=\frac{TP}{TP+FN}\Rightarrow (recall)(TP+FN)=TP\Rightarrow TP(1-recall)=...


5

It's a mistake on Wikipedia. $F_{1}$ as the harmonic mean is defined only at positive real numbers. $PRE$ or $REC$ could be equal 0 in case $TP=0$. Which provides to undefined result $F_1=\frac{0}{0}$.


4

I do not know a standard answer to this, but I thought about it some times ago and I have some ideas to share. When you have one confusion matrix, you have more or less a picture of how you classification model confuse (mis-classify) classes. When you repeat classification tests you will end up having multiple confusion matrices. The question is how to get ...


4

There are a few ways to achieve your "master confusion matrix". Sum all the confusion matrices together: Like you suggested, summing this results in a confusion matrix. The problem with this is you can not interpret totals. Average the entries. This method is the same as number one, but you divide each entry by the number of trials (~400 in your case). ...


4

sklearn.metrics.classification_report provides precision and recall for all classes along with F-score and support. It might prove to be helpful in your case of 3 classes.


4

A confusion matrix can be used to measure the performance of a particular classifier with a fixed threshold. Given a set of input cases, the classifier scores each one, and score above the threshold are labelled Class 1 and scores below the threshold are labelled Class 2. The ROC curve, on the other hand, examines the performance of a classifier without ...


4

what you encounter are real-world problems rarely taught in classes. For training, I would test SKLearn's class_weight = "balanced" or class_weight={0:0.995, 1:0.005}. It's a very robust technique. For testing, you can't fiddle with the class_weight. It's meant to simulate the real-world data. Make sure you don't overfit. E.G. Decision Tree ...


4

A confusion matrix is a table that is often used to describe the performance of a classification model. The figure you have provided presents a binary case, but it is also used with more than 2 classes (there are just more rows/columns). The rows refer to the actual Ground-Truth label/class of the input and the columns refer to the prediction provided by ...


4

Seems like you understand the meaning of the confusion matrix, but not the logic used to name its entries! Here are my 5 cents: The names are all of this kind: <True/False> <Positive/Negative> | | Part1 Part2 The first part explains if the prediction was right or not. If you have only True Positive and True ...


4

I'll try to answer this with a couple examples: Say we have 100 instances (55 negative, 45 positive). Let's say we predict 1/45 positives and 55/55 negatives correctly. Then our accuracy is 0.56 but our F1 score is 0.0435. Now suppose we predict everything as positive: we get an accuracy of 0.45 and an F1 score of 0.6207. Therefore, accuracy does not have to ...


4

A confusion matrix is indeed a very useful way to analyze the results of your experiment. It provides the exact number (or percentage) of instances with true class X predicted as class Y for all the possible classes. As such it gives a detailed picture of what the system classifies correctly or not. But a confusion matrix is a bit too detailed if one wants ...


3

Create a method that does the printing for you: def print_confusion_matrix(y_true, y_pred): cm = confusion_matrix(y_true, y_pred) print('True positive = ', cm[0][0]) print('False positive = ', cm[0][1]) print('False negative = ', cm[1][0]) print('True negative = ', cm[1][1]) And use it like this print_confusion_matrix(x_test, x_pred) ...


3

I would strongly recommend against using a confusion wheel to visualize your confusion matrix. As impressive and fancy as they look, confusion wheels are visually complicated and unintuitive to read. Good data visualizations summarize information in a way that is simple, clear, and intuitive. Confusion wheels have none of those properties. Unfortunately, ...


3

The confusion matrix is used to tell you how many predictions were classified correctly or incorrectly. You are looking at a regression model, which gives you a continous output (not classification). So when you run confusion_matrix(y_test, y_pred) it will throw the ValueError because it expected class predictions, not floating point numbers. Are you ...


3

Please find the below: False Negative (FN): prediction is NEGATIVE, actual outcome is POSITIVE, result is 'False Negative' - Why is that? Shouldn't it be 'False Positive'? Answer : The predictive model supposed to give the answer as 'Positive', but it predicted as 'Negative', which means Falsely predicted as Negative aka False Negative. False Positive (FP):...


3

Thanks for clear statement of the problem. The point is that if you want to decrease false negatives, you should sufficiently lower the threshold of your decision function. If the false negatives are decreased, as you mentioned, true positives increase but false positives can also increase. As a result, recall will increase and precision will decrease.


3

You are correct @Tolga, both can increase at the same time. Consider the following data: Prediction | True Class 1.0 | 0 0.5 | 1 0.0 | 0 If you set your cut off point as 0.75, then you have $$ Precision = \frac{TP}{TP + FP} = \frac{0}{0 + 1} = 0 $$ $$ Recall = \frac{TP}{TP + FN} = \frac{0}{0 + 1} = 0$$ then if you decrease your cut ...


3

In a multiclass problem there is one score for each class, counting any other class as a negative. For example for class 1: TP instances are gold standard class 1 predicted as class 1 FN instances are gold standard class 1 predicted as class 2,3 or 4 FP instances are gold standard class 2,3 or 4 predicted as class 1 TN instances are gold standard class 2,...


3

Well, grid search involves finding best hyperparameters. Best according to what data set? a held out validation set. If that's what you mean by cross validation, then they necessarily happen simultaneously. It doesn't really make sense to do something called cross validation before testing hyperparams - indeed, what would you be evaluating? CV as in k-fold ...


Only top voted, non community-wiki answers of a minimum length are eligible