25

Definitions Accuracy: The amount of correct classifications / the total amount of classifications. The train accuracy: The accuracy of a model on examples it was constructed on. The test accuracy is the accuracy of a model on examples it hasn't seen. Confusion matrix: A tabulation of the predicted class (usually vertically) against the actual class (thus ...


17

To get a confusion matrix from the test data you should go througt two steps: Make predictions for the test data For example, use model.predict_generator to predict the first 2000 probabilities from the test generator. generator = datagen.flow_from_directory( 'data/test', target_size=(150, 150), batch_size=16, class_mode=...


12

Multi-class Confusion Matrix is very well established in literature; you could find it easily on your own. Anyhow, Scikit-learn can do it easily like: from sklearn.metrics import confusion_matrix y_true = ['Cat', 'Dog', 'Rabbit', 'Cat', 'Cat', 'Rabbit'] y_pred = ['Dog', 'Dog', 'Rabbit', 'Dog', 'Dog', 'Rabbit'] classes=['Cat', 'Dog', 'Rabbit'] ...


10

Considering you have two lists y_actual and y_pred ( I assume you made a typo error on x_test and x_pred as in your code), you can pass the two lists to this function to parse them def perf_measure(y_actual, y_pred): TP = 0 FP = 0 TN = 0 FN = 0 for i in range(len(y_pred)): if y_actual[i]==y_pred[i]==1: TP += 1 ...


8

You are just predicting if Play = Yes or Play = No. The confusion matrix would look like this: Predicted +------+------+ | Yes | No | +-------------------+ A | | | | c | Yes | TP | FP | t | | | | u +-------------------+ a | | | | l | No | FN | TN | | ...


7

You can apply a technique I described in my masters thesis (page 48ff) and called Confusion Matrix Ordering (CMO): Order the columns/rows in such a way, that most errors are along the diagonal. Split the confusion matrix into multiple blocks such that the single blocks can easily printed / viewed - and such that you can remove some of the blocks because ...


7

If we decrease the false negative (select more positives), recall always increases, but precision may increase or decrease. Generally, for models better than random, precision and recall have an inverse relationship (@pythinker's answer), but for models worse than random, they have a direct relationship (@kbrose's example). It is worth noting that we can ...


7

F1 will never be zero, but very near to zero for a bad classifier. If TP or TN is zero then there isn't any need to check F1.


6

what you encounter are real-world problems rarely taught in classes. For training, I would test SKLearn's class_weight = "balanced" or class_weight={0:0.995, 1:0.005}. It's a very robust technique. For testing, you can't fiddle with the class_weight. It's meant to simulate the real-world data. Make sure you don't overfit. E.G. Decision Tree ...


5

Maybe not the answer you would like to hear, but I have to admit that I find wikipedia page on the topic definitively well done... https://en.wikipedia.org/wiki/Precision_and_recall


5

I do not know a standard answer to this, but I thought about it some times ago and I have some ideas to share. When you have one confusion matrix, you have more or less a picture of how you classification model confuse (mis-classify) classes. When you repeat classification tests you will end up having multiple confusion matrices. The question is how to get ...


5

Adding to the answer above, The labeling totally depends on how you define it. You can define 0 as negative or as positive. However, for the sake of understanding and ease of readability, keep it meaningful. The instances that are correctly predicted are given by the diagonal. Here, '1' is True Negative or for the class labelled as 0 and '5' is True ...


5

Yes and No! depending on what do you mean by minimization. When you say minimizing $f$ and $g$ according to something, you are actually looking for a point which minimizes both. It does not mean that this point necessarily finds the minimum of $f$ or $g$. So yes in this sense. But in case you mean a point in which both of them are in their minimum, this ...


5

You can! The trick is that you actually know two other critical variables: the number of positive and negative examples (P and N). You can then use them to algebraically solve for the confusion matrix: $ recall=\frac{TP}{TP+FN}=1-\frac{FN}{P}\Rightarrow $ $ FN = P(1-recall) $ $ recall=\frac{TP}{TP+FN}\Rightarrow (recall)(TP+FN)=TP\Rightarrow TP(1-recall)=...


5

It's a mistake on Wikipedia. $F_{1}$ as the harmonic mean is defined only at positive real numbers. $PRE$ or $REC$ could be equal 0 in case $TP=0$. Which provides to undefined result $F_1=\frac{0}{0}$.


5

CM is about True Vs Predicted. Since only one node is in the discussion which means we will have only one column of "Predicted" but all the possible rows of "True" Let's assume 100 samples in the Node. The Node must be classified as "A" Below should be the CM - Rows are "True" and the Column is the "Predicted"...


4

Create a method that does the printing for you: def print_confusion_matrix(y_true, y_pred): cm = confusion_matrix(y_true, y_pred) print('True positive = ', cm[0][0]) print('False positive = ', cm[0][1]) print('False negative = ', cm[1][0]) print('True negative = ', cm[1][1]) And use it like this print_confusion_matrix(x_test, x_pred) ...


4

There are a few ways to achieve your "master confusion matrix". Sum all the confusion matrices together: Like you suggested, summing this results in a confusion matrix. The problem with this is you can not interpret totals. Average the entries. This method is the same as number one, but you divide each entry by the number of trials (~400 in your case). ...


4

I would strongly recommend against using a confusion wheel to visualize your confusion matrix. As impressive and fancy as they look, confusion wheels are visually complicated and unintuitive to read. Good data visualizations summarize information in a way that is simple, clear, and intuitive. Confusion wheels have none of those properties. Unfortunately, ...


4

sklearn.metrics.classification_report provides precision and recall for all classes along with F-score and support. It might prove to be helpful in your case of 3 classes.


4

A confusion matrix can be used to measure the performance of a particular classifier with a fixed threshold. Given a set of input cases, the classifier scores each one, and score above the threshold are labelled Class 1 and scores below the threshold are labelled Class 2. The ROC curve, on the other hand, examines the performance of a classifier without ...


4

The confusion matrix is used to tell you how many predictions were classified correctly or incorrectly. You are looking at a regression model, which gives you a continous output (not classification). So when you run confusion_matrix(y_test, y_pred) it will throw the ValueError because it expected class predictions, not floating point numbers. Are you ...


4

A confusion matrix is a table that is often used to describe the performance of a classification model. The figure you have provided presents a binary case, but it is also used with more than 2 classes (there are just more rows/columns). The rows refer to the actual Ground-Truth label/class of the input and the columns refer to the prediction provided by ...


4

Seems like you understand the meaning of the confusion matrix, but not the logic used to name its entries! Here are my 5 cents: The names are all of this kind: <True/False> <Positive/Negative> | | Part1 Part2 The first part explains if the prediction was right or not. If you have only True Positive and True ...


4

I'll try to answer this with a couple examples: Say we have 100 instances (55 negative, 45 positive). Let's say we predict 1/45 positives and 55/55 negatives correctly. Then our accuracy is 0.56 but our F1 score is 0.0435. Now suppose we predict everything as positive: we get an accuracy of 0.45 and an F1 score of 0.6207. Therefore, accuracy does not have to ...


4

A confusion matrix is indeed a very useful way to analyze the results of your experiment. It provides the exact number (or percentage) of instances with true class X predicted as class Y for all the possible classes. As such it gives a detailed picture of what the system classifies correctly or not. But a confusion matrix is a bit too detailed if one wants ...


3

Please find the below: False Negative (FN): prediction is NEGATIVE, actual outcome is POSITIVE, result is 'False Negative' - Why is that? Shouldn't it be 'False Positive'? Answer : The predictive model supposed to give the answer as 'Positive', but it predicted as 'Negative', which means Falsely predicted as Negative aka False Negative. False Positive (FP)...


3

Thanks for clear statement of the problem. The point is that if you want to decrease false negatives, you should sufficiently lower the threshold of your decision function. If the false negatives are decreased, as you mentioned, true positives increase but false positives can also increase. As a result, recall will increase and precision will decrease.


3

You are correct @Tolga, both can increase at the same time. Consider the following data: Prediction | True Class 1.0 | 0 0.5 | 1 0.0 | 0 If you set your cut off point as 0.75, then you have $$ Precision = \frac{TP}{TP + FP} = \frac{0}{0 + 1} = 0 $$ $$ Recall = \frac{TP}{TP + FN} = \frac{0}{0 + 1} = 0$$ then if you decrease your cut ...


3

In a multiclass problem there is one score for each class, counting any other class as a negative. For example for class 1: TP instances are gold standard class 1 predicted as class 1 FN instances are gold standard class 1 predicted as class 2,3 or 4 FP instances are gold standard class 2,3 or 4 predicted as class 1 TN instances are gold standard class 2,...


Only top voted, non community-wiki answers of a minimum length are eligible