Skip to main content

Questions tagged [metric]

A metric is a way to evaluate the performance of a machine learning model. Depending on the task, different metrics may be used.

Filter by
Sorted by
Tagged with
44 votes
6 answers
63k views

What is the relationship between the accuracy and the loss in deep learning?

I have created three different models using deep learning for multi-class classification and each model gave me a different accuracy and loss value. The results of the testing model as the following: ...
N.IT's user avatar
  • 1,995
30 votes
2 answers
67k views

How to interpret classification report of scikit-learn?

As you can see, it is about a binary classification with linearSVC. The class 1 has a higher precision than class 0 (+7%), but class 0 has a higher recall than class 1 (+11%). How would you interpret ...
user avatar
22 votes
1 answer
25k views

What's the difference between Sklearn F1 score 'micro' and 'weighted' for a multi class classification problem?

I have a multi-class classification problem with class imbalance. I searched for the best metric to evaluate my model. Scikit-learn has multiple ways of calculating the F1 score. I would like to ...
Fractale's user avatar
  • 345
15 votes
3 answers
21k views

MAD vs RMSE vs MAE vs MSLE vs R²: When to use which?

In regression problems, you can use various different metrics to check how well your model is doing: Mean Absolute Deviation (MAD): In $[0, \infty)$, the smaller the better Root Mean Squared Error (...
Martin Thoma's user avatar
13 votes
1 answer
9k views

Balanced Accuracy vs. F1 Score

I've read plenty of online posts with clear explanations about the difference between accuracy and F1 score in a binary classification context. However, when I came across the concept of balanced ...
Ric S's user avatar
  • 267
13 votes
3 answers
2k views

Why is the F-measure preferred for classification tasks?

Why is the F-measure usually used for (supervised) classification tasks, whereas the G-measure (or Fowlkes–Mallows index) is generally used for (unsupervised) clustering tasks? The F-measure is the ...
Bruno Lubascher's user avatar
12 votes
4 answers
16k views

Can the F1 score be equal to zero?

As it is mentioned in the F1 score Wikipedia, 'F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0'. What is the worst condition that was mentioned? Even if we ...
akhil penta's user avatar
12 votes
1 answer
592 views

Finding linear transformation under which distance matrices are similar

I have $n$ sets of vectors, where each set $S_i$ contains $k$ vectors in $\mathbb{R}^d$. I know there is some unknown linear transformation $W$ under which the distance matrix $D_i$ (a $k\times k$ ...
user1767774's user avatar
10 votes
4 answers
9k views

Log loss vs accuracy for deciding between different learning rates?

While model tuning using cross validation and grid search I was plotting the graph of different learning rate against log loss and accuracy separately. Log loss When I used log loss as score in ...
CodeMaster GoGo's user avatar
10 votes
1 answer
3k views

XGBoost custom objective for regression in R

I implemented a custom objective and metric for a xgboost regression. In order to see if I'm doing this correctly, I started with a quadratic loss. The ...
Peter's user avatar
  • 7,526
9 votes
5 answers
14k views

Cosine similarity vs The Levenshtein distance

I wanted to know what is the difference between them and in what situations they work best? As per my understanding: Cosine similarity is a measure of similarity between two non-zero vectors of an ...
Pluviophile's user avatar
  • 3,918
9 votes
3 answers
18k views

Why do we use a Gaussian kernel as a similarity metric?

In graph-based clustering, why is it preferred to use the Gaussian kernel rather than the distance between two points as the similarity metric?
zfb's user avatar
  • 91
8 votes
3 answers
14k views

Good performance metrics for multiclass classification problem besides accuracy?

I am trying to solve a multiclass classification problem. The dataset is balanced. I have been using accuracy as a performace metric till now. Are there any other good performance metrics for this ...
Kishan Kumar's user avatar
8 votes
2 answers
13k views

AUC-ROC for Multi-Label Classification

Hey guys I'm currently reading about AUC-ROC and I have understood the binary case and I think that I understand the multi-classification case. Now I'm a bit confused on how to generalize it to the ...
NotoriousFunk's user avatar
8 votes
3 answers
729 views

Chi-square as evaluation metrics for nonlinear machine learning regression models

I am using machine learning models to predict an ordinal variable (values: 1,2,3,4, and 5) using 7 different features. I posed this as a regression problem, so the final outputs of a model are ...
Alex's user avatar
  • 181
7 votes
1 answer
17k views

What is Continuous Ranked Probability Score (CRPS)?

I came across some evolution metric at Kaggle: Continuous Ranked Probability Score (CRPS): Mathematically, $C = \frac{1}{199N} \sum_{m=1}^{N} \sum_{n=-99}^{99} (P(y \le n) -H(n - Y_m))^2,$ where P ...
user avatar
6 votes
5 answers
27k views

How to compute f1 in TensorFlow

I have a code that computes the accuracy, but now I would like to compute the F1 score. ...
William Scott's user avatar
6 votes
3 answers
353 views

How to get K most different rows in csv?

We have boring CSV with 10000 rows of ages (float), titles (enum/int), scores (float). How to select 1000 most different rows? I look for a general solution that ...
Blender's user avatar
  • 161
6 votes
1 answer
736 views

F - measure derivation (harmonic mean of precision and recall)

We can define the F - measure as follows: $F_{\alpha}=\frac{1}{\alpha \frac{1}{P}+(1-\alpha)\frac{1}{R}} $ Now we might be interested in choosen a good $\alpha$. In the article The truth of the ...
Patricio's user avatar
  • 163
6 votes
2 answers
3k views

issue with early-stopping on f1 score with imbalanced data

I have a highly imbalanced dataset with less than 0.5% of the minor class. Using Keras, I'm training DNN on the training set and evaluate performance on validation set. Loss function is ...
zesla's user avatar
  • 181
5 votes
1 answer
5k views

F1_score(average='micro') is equal to calculating accuracy for multiclasification

Is f1_score(average='micro') always the same as calculating the accuracy. Or it is just in this case? I have tried with different values and they gave the same answer but I don't have the analytical ...
Carlos Mougan's user avatar
5 votes
1 answer
8k views

Inception Score (IS) and Fréchet Inception Distance (FID), which one is better for GAN evaluation?

IS uses two criteria in measuring the performance of GAN: The quality of the generated images, and their diversity based on the entropy of the distribution of synthetic data. On the other hand, FID ...
Giang Nguyen's user avatar
5 votes
1 answer
4k views

What is Bit Per Character?

What is Bits per Character (bpc) metric which has been used to measure the model accuracy with reference to text8 and ...
Ashwin Geet D'Sa's user avatar
5 votes
1 answer
2k views

Loss function for optimising precision & recall / sensitivity & specificity?

I've been using precision and recall as my metrics, as per keras-team/keras/pull/9393/files Sensitivity & specificity is what I want to optimise for. Every epoch I output it: ...
A T's user avatar
  • 171
4 votes
1 answer
4k views

Interpretability of RMSE and R squared scores on cross validation

I'm working on a regression problem with 30k rows in my dataset, decided to use XGBoost mainly to avoid processing data for a quick primitive model. And i noticed upon doing cross-validation that ...
Blenz's user avatar
  • 2,084
4 votes
3 answers
889 views

Which metric to use for imbalanced data in TensorFlow/Keras

I am doing a binary classification task with Keras and my model directly outputs either 0 or 1. Typically I compile the model like something below: ...
D.J. Elkind's user avatar
4 votes
3 answers
419 views

A metric between trees

I have certain tree structures. I am not an expert in machine learning. As I would with take KNN, I would calculate distances via metric function and a new data point and the points from the training ...
Chris Pillen's user avatar
4 votes
1 answer
1k views

Choose ROC/AUC vs. precision/recall curve?

I am trying to get a clear understanding on various classification metrics, including knowing when to choose ROC/AUC as opposed to opting for the Precision/Recall curve. I am reading Aurélien Géron's ...
lazarea's user avatar
  • 299
4 votes
0 answers
233 views

Fast PR / ROC curves and corespondings AUPR / AUROC

I find myself in a position of calculating numerous PR / ROC curves and their associated area under the PR curves (AUPR) / area under the ROC curve (AUROC). Its is quite easy to perform those ...
Lucas Morin's user avatar
  • 2,289
3 votes
3 answers
2k views

Is R2 score a reasonable regression measure on huge datasets?

I'm running a regression model on a pretty large data set and getting a fairly woeful $R^2$ score of ~0.2 (see plot below), despite the plot looking like the model is generally pointing in the right ...
jshep's user avatar
  • 393
3 votes
3 answers
588 views

What is AUC - ROC Curve?

AUC - ROC curve is a performance measurement for classification problem at various thresholds settings. ROC is a probability curve and AUC represents degree or measure of separability. Is Roc the ...
user avatar
3 votes
2 answers
1k views

Precision, recall and importance of them in the imbalance problem

I have a test dataset. The dataset is an imbalanced dataset. The total training instances for the dataset is 543 among them minority class(yes) is 75 and the majority class(No) is 468. The class of ...
Encipher's user avatar
  • 361
3 votes
1 answer
1k views

How to select 'cutoff' of classifier probability

I have recently used xgboost to conduct binary classification in an nlp problem. The idea was to identify if a particular article belonged to an author or not, pretty standard exercise. The results ...
bls's user avatar
  • 143
3 votes
2 answers
185 views

Is it correct to define the F-measure as the harmonic mean of specificity and sensitivity in such a way?

It is common to define the F-measure as a function of precision and recall, as mentioned in [1]: $F_{\beta}=\frac{(1+\beta^2)PR}{\beta^2 P+R}$ However I came across some other cases, another ...
Qubit's user avatar
  • 33
3 votes
1 answer
4k views

Accuracy vs Categorical Accuracy

I was running a DNN model that uses ResNet50 for Transfer Learning. While fitting the training data on my model to check the initial trend (would run for more epochs if initial trend seems right), I ...
Harsh Khare's user avatar
3 votes
1 answer
2k views

scikit-learn classification report's f1 accuracy?

When I run scikit-learn classification_report() on my 2-class y and yhat, I get the ...
stackoverflowuser2010's user avatar
3 votes
1 answer
978 views

Can macro F1 score be greater than micro F1 score?

I am reading about evaluation metrics, and it seems that micro scores are more useful. But I was wondering about scenarios where macro F1 score is greater than micro F1 score, and if this is at all ...
user avatar
3 votes
1 answer
91 views

What does precision-recall curve and ROC curve tell us abouth threshold invariance

Consider a binary classification problem. Intuitively, a value for the area under the curve (for both curves) very close to 1, shows that the curve is almost L-shaped. Thus, this means that the value ...
liakoyras's user avatar
  • 636
3 votes
2 answers
4k views

How to compute G-mean score?

I would greatly appreciate if you could let me know how to fix the following issue: I used sklearn.metrics.fowlkes_mallows_score to compute G-mean score for my binary classification problem, but it ...
ebrahimi's user avatar
  • 1,307
3 votes
1 answer
55 views

Classification problem: custom minimization measure

Assume a binary classification problem, with $1$ denoted as a "bad" outcome, and $0$ as a "good" outcome. If it's relevant, in the sample there are significantly more bads than goods. I'm trying to ...
runr's user avatar
  • 236
3 votes
2 answers
6k views

Calculating an estimate of KL Divergence using the samples drawn from distributions

Given two sets of samples drawn from two different distributions, is it computationally possible to get an estimate of KL-Divergence between the two distribution using these samples? Here I am ...
JVG's user avatar
  • 31
3 votes
2 answers
399 views

How to measure Entity Ambiguity?

When using/building a system for Entity Linking, is there a well-known measure for "ambiguity degree" of an entity? Some approach to compare named entities regarding how difficult to disambiguate?
Abdulrahman Bres's user avatar
3 votes
1 answer
1k views

What is the correct formula for Jaccard coefficient with integer vectors?

I understand the Jaccard index is the number of elements in common divided by the total number of distinct elements. But it seems to be some discrepancy or terminology confusion about Jaccard being ...
Veronica's user avatar
  • 133
3 votes
0 answers
339 views

Why is NDCG high even for wrongly ranked predictions?

The NDCG (Normalized Discounted Cumulative Gain) metric for ranking is defined as DCG/IDCG, where IDCG is the ideal DCG and is said to take values in [0, 1]. However, since the DCG will always be ...
Michael's user avatar
  • 131
3 votes
0 answers
93 views

How do I make inference about test metrics for entire population from sample metrics?

Generally we calculate specific metrics for ML models on a test set (and we try to make that test set representative). I'm not clear on how to make inference about the same metrics for the population ...
Shirish's user avatar
  • 299
3 votes
0 answers
537 views

Keras custom metrics - MAP and MRR

I am trying to build a LSTM model in keras where I have one question with 10 answers but only ONE among them is correct. So basically im tring to build a 10 class classification problem. As most of ...
Rohith's user avatar
  • 31
3 votes
0 answers
471 views

What is the difference between KL-divergence, JS-divergence, Wasserstein distance and MMD?

I was reading about different distribution distances, and came across Kullback-Leibler divergence Jensen-Shannon divergence Wasserstein distance Maximum mean discrepancy (MMD) The book was too ...
asahi kibou's user avatar
2 votes
2 answers
1k views

Some simple questions about confusion matrix and metrics in general

I will first tell you about the context then ask my questions. The model detects hate speech and the training and testing datasets are imbalanced (NLP). My questions: Is this considered a good model? ...
Maxi's user avatar
  • 89
2 votes
2 answers
114 views

Choosing best model produced from different algorithms. Metric produced by cross-validation on the train set or metric produced on the test set?

I know that choosing between models produced by one algorithm with different hyperparameters the metric for choosing the best one should be the cross-validation on train set. But what about choosing ...
vasili111's user avatar
  • 157
2 votes
1 answer
573 views

Confusion between precision and recall

I have a machine learning model that try to fingerprint the functions in a binary file with a corpus. Final output of upon inputing a binary file is a table with one to one mapping between the binary ...
hEShaN's user avatar
  • 131

1
2 3 4 5