# Questions tagged [metric]

The tag has no usage guidance.

107 questions
Filter by
Sorted by
Tagged with
35 views

### What is evaluation metric for two sets? [closed]

I've two sets one is ground truth and other is output of my machine learning models. Assume my groundtruth set is A={1,2,3,4,5} and output of machine learning model is B={3,4,5,6,7,8}. One way I can ...
10 views

### What is the state of the art/research metric to compare ellipses but jaccard coefficient?

Im looking for the, if there is one, metric to compare ellipses with each other. Last time a had a similar dataset (malaria cells, now its pupiles) i used jaccard coefficient but that was more because ...
24 views

### Loss and Metrics for COCO Keypoints

I am using Keras to train different models on the COCO keypoints dataset. All of the models I am working with are used for image segmentation, so they output heatmaps corresponding to the labels. All ...
19 views

### Inception Score (IS) and Fréchet Inception Distance (FID), which one is better for GAN evaluation?

IS uses two criteria in measuring the performance of GAN: The quality of the generated images, and their diversity based on the entropy of the distribution of synthetic data. On the other hand, FID ...
14 views

### Is there a way to normalize a similarity matrix by row and column in a way such that only one entry per row or column is approximately 1

I am computing similarities between 2 vectors. My goal is to have approximately 1 matching sample with similarity ~1, for each sample, without having any samples that are similar to many other samples....
27 views

### Is it correct to define the F-measure as the harmonic mean of specificity and sensitivity in such a way?

It is common to define the F-measure as a function of precision and recall, as mentioned in : $F_{\beta}=\frac{(1+\beta^2)PR}{\beta^2 P+R}$ However I came across some other cases, another ...
42 views

### AUC-ROC for Multi-Label Classification

Hey guys I'm currently reading about AUC-ROC and I have understood the binary case and I think that I understand the multi-classification case. Now I'm a bit confused on how to generalize it to the ...
18 views

### Recall score for each sample in multilabel classification

Does it make sense to calculate the recall for each sample in a multilabel classification problem? Suppose I have 3 data samples, each having its own true set of labels and predicted set of labels. ...
25 views

### How to re-train a model from false positives

I'm still a bit new to deep learning. What I'm still struggling, is what is the best practice in re-training a good model over time? I've trained a deep model for my binary classification problem (...
24 views

### what metrics to evaluate rank order results?

I have searched on stackexchange and found a couple of topics like this and this but they are not quite relevant to my problem (or at least I don't know how to make them relevant to my problem). ...
18 views

### Metrics - multi-class model comparisons

I am looking for a way to quantify the performance of multi-class model labelers, and thus compare them. I want to account for the fact that some classes are ‘closer’ than others (for example a car is ...
47 views

### Keras custom metrics - MAP and MRR

I am trying to build a LSTM model in keras where I have one question with 10 answers but only ONE among them is correct. So basically im tring to build a 10 class classification problem. As most of ...
29 views

### How to estimate the accuracy on a large dataset?

Given that I have a deep learning model(handover from former colleague). For some reason, the train/dev set was missing. In my situation, I want to classify my dataset into 100 categories. The ...
13 views

### goodness of fit metrics to compare neural network and GLM model for count data

I´m wodering if some of you have compared goodness of fit of a NN and a GLM model on count data and which metrics you used? In addition, the data I´m dealing with has a point mass at zero. Are there ...
16 views

### What is ERR@K metric and how to interpret it? [closed]

What is ERR@K (e.g. K=10)? what does this metric's value tell me?
21 views

### How to calculate precision and recall?

There is a class-imbalanced labeled dataset with 100'000 samples. 90'000 is "0" and 10'000 is "1". There is a model that predicts the labels. It was runned on the class-balanced (10'000 of "0" and 10'...
67 views

### What is AUC - ROC Curve?

AUC - ROC curve is a performance measurement for classification problem at various thresholds settings. ROC is a probability curve and AUC represents degree or measure of separability. Is Roc the ...
127 views

### What's a good F1-score in (not) extremely imbalanced dataset?

I have a dataset with around 4.7K focused on binary classification. Class proportion is 33:67. meaning Label 1 is 1558 (33%) and Label 0 is 3154 (67%) of my dataset. Is my dataset imbalanced? some ...
66 views

108 views

### How to define quadratic weighted kappa as eval_metric in catboost classifier

I am using catboost for a multiclass classification problem. I want to use quadratic weighted kappa as the evaluation metric. Catboost already has WKappa as an eval_metric but it is linearly weighted ...
331 views

### How do I use Cohen's Kappa as a metric in a keras model for continuous output?

I am using keras to train a CNN on a dataset. The output should be a value from 0 to 4 and my training y values are integers in that range. I want to create a custom metric that rounds and clips my ...
27 views

### How to control the amount of positives in classification?

I have a basic, yet quite complex problem to solve right now. Let's say we have a training set of 20,000 samples in my training set, out of which 3 to 4% is flagged as "True", the rest is flagged as "...
31 views

### What is the meaning of the parameter “metrics” in the method model.compile in Keras?

I don't have very clear the meaning of the parameter metrics of the compile method of the class model in Keras: ...
39 views

### Metric/loss for bin classification

I have a model that has to classify inputs into one of 45 categories but those categories actually represent bins (e.g. bins 1, 2 and 3 are between 1 and 10, 11 and 20, 21 and 30 respectively). What I ...
18 views

### Book about distances for data science (can't remember the name)

I saw a book somewhere that listed a big amount of mathematical distance functions (the usual euclidean norms, the discrete distance, the hamming distance, etc) used for data science. The name of the ...
6 views

### What could be a Replacement Level to calculate some sort of a Wins Above Replacements for an Ecommerce business

I'm trying to create a metric for an ecommerce business. Just like Wins Above Replacements is a metric used in baseball to assess the achievements of a player as compared to a standard replacement ...
35 views

### Sanity check: low PPV but high AUC scores?

I have two algorithms running on a piece of data, both of which perform differently. One of them (call it A) consistently gets a positive predictive value of about 0.75-0.78. Looking at the AUC of ...
63 views

### It seems that the output of sklearn.metrics.pairwise.euclidean_distances is different to the formula on doc

The doc of sklearn.metrics.pairwise.euclidean_distances() gives this formula dist(x, y) = sqrt(dot(x, x) - 2 * dot(x, y) + dot(y, y)). Apply this formula to ...
45 views

### A metric to include both accuracy and time

Problem Statement : Given a signal, predict some property of the signal. Let's say for discussion here that this property is the frequency of the signal. Clearly the output will be a regression value ...
36 views

### Binary classifier on imbalanced dataset yields weird PR curve

I have a dataset with ~6M points, 9 features and two classes. The minority class represents just under 2% of the data. The data is first divided into 100 batches and a different classifier is trained ...
122 views

### Choice of f1 score for highly imbalanced dataset?

I am confused whether to use f1 score with 'micro' average or 'macro' average for better evaluation. Given my dataset is highly imbalanced(600:100000)
68 views

### Ball Tree and Pseudometrics

The docs for sklearn.neighbors.DistanceMetric state that in order to be used within the BallTree, the distance must be a true metric (i.e. be non-negative, 0 ...
15 views

### Appropriate model metric for a truncated response variable?

Here's a straightforward question I can't seem to find a good answer to. Let's say you're using some variables to predict age. I'm assuming a regression model is the right approach. In this case, what ...
34 views

### GridSearch on imbalanced multi-class dataset

I have an imbalanced multi-class dataset (GTSRB) and would like to use GridSearch to determine the hyperparameters for an SVM. As metric for the evaluation I chose F1 with average macro. ...
223 views

### Interpretability of RMSE and R squared scores on cross validation

I'm working on a regression problem with 30k rows in my dataset, decided to use XGBoost mainly to avoid processing data for a quick primitive model. And i noticed upon doing cross-validation that ...
248 views

### Finding linear transformation under which distance matrices are similar

I have n sets of vectors, where each set S_i contains k vectors in ...
43 views

### What is Bit Per Character?

What is Bits per Character (bpc) metric which has been used to measure the model accuracy with reference to text8 and ...
13 views

### what is the representation/meaning/implication in real life of $P(\text{+})$ in the wiki Drug testing Example about Bayes' theorem

In probability theory and statistics, Bayes' theorem (alternatively Bayes' law or Bayes' rule) describes the probability of an event, based on prior knowledge of conditions that might be related to ...