Questions tagged [metric]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2
votes
1answer
19 views

Calculating Rank Ordering Error Metric for implicit recommendation

I'm reading Collaborative Filtering for Implicit Feedback Datasets. On page 6 they detail their evaluation strategy, which they define as mean Expected Percentile Ranking with the following formula: $...
-1
votes
1answer
11 views

F1_score(average='micro') is equal to calculating accuracy for multiclasification

Is f1_score(average='micro') always the same as calculating the accuracy. Or it is just in this case? I have tried with different values and they gave the same answer but I don't have the analytical ...
0
votes
0answers
10 views

Rank impact of individual metrics on overall sales performance

I’m interested in learning about different methods to analyse sales data - at both store level and corporate. Essentially I have lots of performance metrics (~20) that break down the overall sales ...
0
votes
0answers
15 views

Tensorflow API: What does the metric `tf.keras.metrics.TopKCategoricalAccuracy` do?

According to the API doc, this metric "Computes how often targets are in the top K predictions." But how come the following codes prouce the result 1? 0.95>0.9>0.8>0.1>0.05, both 0.95 and 0.8 lead ...
4
votes
1answer
94 views

How to interpret classification report of scikit-learn?

As you can see, it is about a binary classification with linearSVC. The class 1 has a higher precision than class 0 (+7%), but class 0 has a higher recall than class 1 (+11%). How would you interpret ...
1
vote
0answers
18 views

How to create an roc plot and calculate AUC for an svm (that does not return probabilities)?

I have some SVM classifier outputting final classifications for every sample in the test set, something like 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1 and so on. The "...
1
vote
1answer
122 views

What is cohen kappa metric, implementation in Python?

Can somebody explain indetail explanation on Quadratic Kappa Metric/cohen kappa metric with implementation in Python
0
votes
1answer
30 views

What is Continuous Ranked Probability Score (CRPS)?

I came across some evolution metric at Kaggle: Continuous Ranked Probability Score (CRPS): Mathematically, $C = \frac{1}{199N} \sum_{m=1}^{N} \sum_{n=-99}^{99} (P(y \le n) -H(n - Y_m))^2,$ where P ...
0
votes
1answer
17 views

What is the difference between an RMSE and RMSLE (logarithmic error)?

RMSE vs RMSLE Root Mean Squared Error (RMSE) and Root Mean Squared Logarithmic Error (RMSLE) both are the techniques to find out the difference between the values predicted by the machine learning ...
0
votes
3answers
39 views

Cosine similarity vs The Levenshtein distance

Cosine similarity vs The Levenshtein distance I wanted to know what is the difference between them and in what situations they work best? As per my understanding: Cosine similarity is a measure of ...
0
votes
0answers
11 views

Categorical loss functions with similar properties to Kullback-Leibler loss function

When using the Kullback-Leibler divergence as loss function for predicting the probabilities of a categorical (multinomial) distribution, one of the properties is that the difference between $a$ and $...
0
votes
0answers
43 views

How to define quadratic weighted kappa as eval_metric in catboost classifier

I am using catboost for a multiclass classification problem. I want to use quadratic weighted kappa as the evaluation metric. Catboost already has WKappa as an eval_metric but it is linearly weighted ...
1
vote
0answers
110 views

How do I use Cohen's Kappa as a metric in a keras model for continuous output?

I am using keras to train a CNN on a dataset. The output should be a value from 0 to 4 and my training y values are integers in that range. I want to create a custom metric that rounds and clips my ...
1
vote
1answer
26 views

How to control the amount of positives in classification?

I have a basic, yet quite complex problem to solve right now. Let's say we have a training set of 20,000 samples in my training set, out of which 3 to 4% is flagged as "True", the rest is flagged as "...
1
vote
1answer
21 views

What is the meaning of the parameter “metrics” in the method model.compile in Keras?

I don't have very clear the meaning of the parameter metrics of the compile method of the class model in Keras: ...
1
vote
1answer
25 views

Metric/loss for bin classification

I have a model that has to classify inputs into one of 45 categories but those categories actually represent bins (e.g. bins 1, 2 and 3 are between 1 and 10, 11 and 20, 21 and 30 respectively). What I ...
1
vote
1answer
18 views

Book about distances for data science (can't remember the name)

I saw a book somewhere that listed a big amount of mathematical distance functions (the usual euclidean norms, the discrete distance, the hamming distance, etc) used for data science. The name of the ...
0
votes
0answers
6 views

What could be a Replacement Level to calculate some sort of a Wins Above Replacements for an Ecommerce business

I'm trying to create a metric for an ecommerce business. Just like Wins Above Replacements is a metric used in baseball to assess the achievements of a player as compared to a standard replacement ...
0
votes
1answer
16 views

Sanity check: low PPV but high AUC scores?

I have two algorithms running on a piece of data, both of which perform differently. One of them (call it A) consistently gets a positive predictive value of about 0.75-0.78. Looking at the AUC of ...
1
vote
1answer
30 views

It seems that the output of sklearn.metrics.pairwise.euclidean_distances is different to the formula on doc

The doc of sklearn.metrics.pairwise.euclidean_distances() gives this formula dist(x, y) = sqrt(dot(x, x) - 2 * dot(x, y) + dot(y, y)). Apply this formula to ...
1
vote
0answers
43 views

A metric to include both accuracy and time

Problem Statement : Given a signal, predict some property of the signal. Let's say for discussion here that this property is the frequency of the signal. Clearly the output will be a regression value ...
1
vote
0answers
31 views

Binary classifier on imbalanced dataset yields weird PR curve

I have a dataset with ~6M points, 9 features and two classes. The minority class represents just under 2% of the data. The data is first divided into 100 batches and a different classifier is trained ...
0
votes
1answer
70 views

Choice of f1 score for highly imbalanced dataset?

I am confused whether to use f1 score with 'micro' average or 'macro' average for better evaluation. Given my dataset is highly imbalanced(600:100000)
1
vote
1answer
50 views

Ball Tree and Pseudometrics

The docs for sklearn.neighbors.DistanceMetric state that in order to be used within the BallTree, the distance must be a true metric (i.e. be non-negative, 0 ...
1
vote
1answer
13 views

Appropriate model metric for a truncated response variable?

Here's a straightforward question I can't seem to find a good answer to. Let's say you're using some variables to predict age. I'm assuming a regression model is the right approach. In this case, what ...
0
votes
0answers
27 views

GridSearch on imbalanced multi-class dataset

I have an imbalanced multi-class dataset (GTSRB) and would like to use GridSearch to determine the hyperparameters for an SVM. As metric for the evaluation I chose F1 with average macro. ...
4
votes
1answer
160 views

Interpretability of RMSE and R squared scores on cross validation

I'm working on a regression problem with 30k rows in my dataset, decided to use XGBoost mainly to avoid processing data for a quick primitive model. And i noticed upon doing cross-validation that ...
6
votes
0answers
227 views

Finding linear transformation under which distance matrices are similar

I have n sets of vectors, where each set S_i contains k vectors in ...
1
vote
0answers
16 views

What is Bit Per Character?

What is Bits per Character (bpc) metric which has been used to measure the model accuracy with reference to text8 and ...
0
votes
1answer
12 views

what is the representation/meaning/implication in real life of $P(\text{+})$ in the wiki Drug testing Example about Bayes' theorem

In probability theory and statistics, Bayes' theorem (alternatively Bayes' law or Bayes' rule) describes the probability of an event, based on prior knowledge of conditions that might be related to ...
1
vote
0answers
143 views

What's the difference between these two custom sparse categorical accuracy functions?

I have a sequence classification model featuring CustomELMo Embeddings layer + BiLSTM + Fully Connected layer. I've found two custom metrics for sparse_categorical_accuracy, but can't wrap my head ...
0
votes
0answers
12 views

How to optimize MAPE in regression algorithms

I have a regression task where the label is varying from about 0.001 to 1000. One of the feature called group, for example, group A corresponding label from 0-0.1 and group G corresponding label from ...
0
votes
0answers
17 views

Loss function for multi-class classifiction where output variable is a level i.e the various classes are dependent on each other

Let's say we are classifying Images of cat , fish and human. Classifying a cat as human is as wrong as classifying it as fish, so here the normal loss functions/ metrics like Confusion matrix is fine. ...
0
votes
1answer
31 views

How is the “loss” calculated which is supplied by the callback log in Keras?

I.e. categorical cross entropy? binary cross entropy? Something else? Or is it perhaps the loss function which you pass into the model.compile method?
2
votes
2answers
43 views

Metrics for Name Entity Recognition

Working on a NER project, I have been facing the problem of evaluating my model during training. I cannot be using the accuracy metrics or f1 score or any other metrics to evaluate my model on runtime ...
0
votes
0answers
51 views

For a multiclass classification problem, how do we find the cohen kappa score?

So I have a multiclass classification problem and I have found the Matthews Correlation Coefficient of that (https://scikit-learn.org/stable/modules/model_evaluation.html#matthews-correlation-...
0
votes
1answer
28 views

How is computed the train data set score and why?

In a lot of machine learning blogs or review, the training dataset accuracy (or other metric) is given alongside the test dataset score. Is this score calculated through the training, or is the ...
0
votes
2answers
25 views

I am getting different mean_absolute_error when i retrain my model everything same

I have set my numpy random seed to 0. I am training on colab and using keras. I didn't change anything. I just re-ran my cell and the val_absolute_error changed. Code: ...
0
votes
1answer
104 views

Best metric in imbalanced classification for multi-label classification

My test data are imbalanced, i tried to use the precision or the gmean as metrics for a multi-label learning model, but both metrics are not very informative. Is there any way to use for example the ...
0
votes
0answers
11 views

Manual setting of target variable based on features' minimum values: f1 score = 1

I am building a classifier for user engagement in my website. Basically, since there are no "proxy" for engagement, i.e. there is no pre-defined target variable, I came up with minimum thresholds ...
1
vote
0answers
37 views

What is divergence exactly in machine learning?

I know about KL divergence, JS Divergence and clearly know that it is different from the divergence in calculus. I have an intutive feeling of divergence as it roughly compares the closeness of two ...
1
vote
1answer
65 views

How to interpret metrics of a model after scaling the data

I have a GradientBoostingRegressor from scikit-learn which I trained. Afterwards, I obviously would like to know how good the model is. So, on a non-scaled dataset ...
0
votes
0answers
45 views

I got a overall mean average precision score of 0 for a recommendation engine

I just wanted to know if receiving an overall MAP score of 0 in a recommendation engine was possible, or a sign that my calculation or my logic for the engine was wrong.
0
votes
1answer
65 views

Metric (rather than RMSE, MSE, etc.) to choose the best model in terms of the ability to detect peaks better

I have created multiple regression models and wanted to choose the best one. One common metric would be RMSE, as you know. When I looked at the results, second model (RMSE = 0.15) was better able to ...
0
votes
0answers
15 views

How to measure the success of a seasonal product or what should be the northstar KPI

Not sure if this is the right forum to post this question. This is more of a product management question, I searched all the communities in StackExchange and thought this might be the closest to ...
1
vote
0answers
42 views

Is there a metric for counting AND classification simultaneously?

I'm working on a project that mixes object detection and crowd counting. The metric for object detection is mAP, which combines the regression of the bounding boxes with the precision of the ...
1
vote
3answers
1k views

How to compute f1 in tensorflow

I have code that computes the accuracy, but now I would like to compute the f1 score. ...
2
votes
3answers
121 views

How to select between models when AUC scores are similar?

I use two machine learning algorithms for binary classification and I get this result : Algo 1 : ...
1
vote
0answers
13 views

Metrics for stain normalization

Are there any metrics or methods for assessing stain normalization techniques?
1
vote
0answers
72 views

What does the ratio of precision/recall to their ideal values mean, and why are they equal?

When evaluating rankings, Normalized Discounted Cumulative Gain (NDCG) normalizes the score to a [0,1] range by dividing by the ideal score. What happens if we take the same idea to Precision (Pre) ...