Questions tagged [metric]

The tag has no usage guidance.

0
votes
1answer
11 views

How is the “loss” calculated which is supplied by the callback log in Keras?

I.e. categorical cross entropy? binary cross entropy? Something else? Or is it perhaps the loss function which you pass into the model.compile method?
2
votes
1answer
21 views

Metrics for Name Entity Recognition

Working on a NER project, I have been facing the problem of evaluating my model during training. I cannot be using the accuracy metrics or f1 score or any other metrics to evaluate my model on runtime ...
0
votes
0answers
7 views

For a multiclass classification problem, how do we find the cohen kappa score?

So I have a multiclass classification problem and I have found the Matthews Correlation Coefficient of that (https://scikit-learn.org/stable/modules/model_evaluation.html#matthews-correlation-...
0
votes
1answer
19 views

How is computed the train data set score and why?

In a lot of machine learning blogs or review, the training dataset accuracy (or other metric) is given alongside the test dataset score. Is this score calculated through the training, or is the ...
0
votes
2answers
17 views

I am getting different mean_absolute_error when i retrain my model everything same

I have set my numpy random seed to 0. I am training on colab and using keras. I didn't change anything. I just re-ran my cell and the val_absolute_error changed. Code: ...
0
votes
1answer
29 views

Best metric in imbalanced classification for multi-label classification

My test data are imbalanced, i tried to use the precision or the gmean as metrics for a multi-label learning model, but both metrics are not very informative. Is there any way to use for example the ...
0
votes
0answers
8 views

Manual setting of target variable based on features' minimum values: f1 score = 1

I am building a classifier for user engagement in my website. Basically, since there are no "proxy" for engagement, i.e. there is no pre-defined target variable, I came up with minimum thresholds ...
0
votes
2answers
31 views

Classifier performance metric choice [closed]

I am having a tough time picking the right metric to assess how well my classifier performs. It is a binary classification problem with imbalanced data (90:10). It is important to classify both ...
1
vote
0answers
18 views

What is divergence exactly in machine learning?

I know about KL divergence, JS Divergence and clearly know that it is different from the divergence in calculus. I have an intutive feeling of divergence as it roughly compares the closeness of two ...
1
vote
1answer
32 views

How to interpret metrics of a model after scaling the data

I have a GradientBoostingRegressor from scikit-learn which I trained. Afterwards, I obviously would like to know how good the model is. So, on a non-scaled dataset ...
0
votes
0answers
14 views

I got a overall mean average precision score of 0 for a recommendation engine

I just wanted to know if receiving an overall MAP score of 0 in a recommendation engine was possible, or a sign that my calculation or my logic for the engine was wrong.
0
votes
1answer
37 views

Metric (rather than RMSE, MSE, etc.) to choose the best model in terms of the ability to detect peaks better

I have created multiple regression models and wanted to choose the best one. One common metric would be RMSE, as you know. When I looked at the results, second model (RMSE = 0.15) was better able to ...
0
votes
0answers
8 views

How to measure the success of a seasonal product or what should be the northstar KPI

Not sure if this is the right forum to post this question. This is more of a product management question, I searched all the communities in StackExchange and thought this might be the closest to ...
1
vote
0answers
37 views

Is there a metric for counting AND classification simultaneously?

I'm working on a project that mixes object detection and crowd counting. The metric for object detection is mAP, which combines the regression of the bounding boxes with the precision of the ...
0
votes
2answers
57 views

How to compute f1 in tensorflow

I have code that computes the accuracy, but now I would like to compute the f1 score. ...
2
votes
3answers
98 views

How to select between models when AUC scores are similar?

I use two machine learning algorithms for binary classification and I get this result : Algo 1 : ...
1
vote
0answers
12 views

Metrics for stain normalization

Are there any metrics or methods for assessing stain normalization techniques?
1
vote
0answers
52 views

What does the ratio of precision/recall to their ideal values mean, and why are they equal?

When evaluating rankings, Normalized Discounted Cumulative Gain (NDCG) normalizes the score to a [0,1] range by dividing by the ideal score. What happens if we take the same idea to Precision (Pre) ...
0
votes
0answers
75 views

Using logLoss as metric function for highly unbalanced dataset

ihave an highly unbalanced dataset and the caret pacjage only allows me to select accuracy or kappa as performance metric. Is it correct to use a mlogloss function to compute model performance? Do you ...
1
vote
0answers
27 views

How to Monitor ML Classification Models in production?

I've often heard of measures like Population Stability Index and Characteristic Stability Index. I might be mistaken, but these seem to be more applicable towards looking at the changes in univariate ...
1
vote
0answers
32 views

how to interpret a high AUC value but a low F1 score after upsampling minority class?

I use to gradient boosting to classify my data between default and paid. The data is very imbalanced where default is in minority. The fisrt classification report from sklearn gradient boosting ...
1
vote
1answer
185 views

Can macro F1 score be greater than micro F1 score?

I am reading about evaluation metrics, and it seems that micro scores are more useful. But I was wondering about scenarios where macro F1 score is greater than micro F1 score, and if this is at all ...
2
votes
1answer
14 views

How should multiclass classifier performance be measured when one type of error is preferred over another?

Sorry if this question has been asked before--I am having trouble searching this topic since I'm not sure of my wording. Say you have a classification problem where there are more than two labels ...
1
vote
1answer
94 views

Metrics for presenting RNN/LSTM result

I am working on a two different architecture based on LSTM model to predict the users next action based on the previous actions. I am wondering, what is the best way to present the result? Is it okay ...
2
votes
3answers
43 views

A metric between trees

I have certain tree structures. I am not an expert in machine learning. As I would with take KNN, I would calculate distances via metric function and a new data point and the points from the training ...
0
votes
0answers
26 views

How to measure accuracy of a route prediction

I developed a new route prediction algorithm and I am trying to find a metric that informs on how well a prediction was. This metric is meant to be used offline, meaning that the goal is not to ...
1
vote
2answers
515 views

MAD vs RMSE vs MAE vs MSLE vs R²: When to use which?

In regression problems, you can use various different metrics to check how well your model is doing: Mean Absolute Deviation (MAD): In $[0, \infty)$, the smaller the better Root Mean Squared Error (...
4
votes
3answers
647 views

What is the relationship between the accuracy and the loss in deep learning?

I have created three different models using deep learning for multi-class classification and each model gave me a different accuracy and loss value. The results of the testing model as the following: ...
0
votes
1answer
46 views

Understanding the Gini/AUC metric as out-of-development performance metric

Assume we develop a model for a binary classification task that reaches a certain Gini/AUROC estimate on the validation ( or training ) sample, among others. This is an overall good metric, often used ...
2
votes
0answers
24 views

Dimension of the manifold on which my data sits

Suppose that I have data points, in the form of vectors with binary entries. We create a metric space, or Vietoris-Rips complex, using the Hamming distance between the data points. I would like to ...
4
votes
1answer
181 views

issue with early-stopping on f1 score with imbalanced data

I have a highly imbalanced dataset with less than 0.5% of the minor class. Using Keras, I'm training DNN on the training set and evaluate performance on validation set. Loss function is ...
3
votes
1answer
2k views

What's the difference between Sklearn F1 score 'micro' and 'weighted' for a multi class classification problem?

I have a multi-class classification problem with class imbalance. I search the best metric to evaluate my model. Sklearn has multiple way of calculating F1 score. I would like to understand the ...
0
votes
1answer
98 views

How do you measure performance for word prediction tasks?

Say I have to predict the next word in a sentence, given the initial few words. Suppose the prefix is "I went to _____". This prefix is common enough that it might appear 10 times in the training ...
1
vote
2answers
165 views

Can Precision-Recall be improved for imbalanced sample?

I tried out a few models on a highly imbalanced sample (~2:100) where I can get decent AUC from ROC (test sample). But when I plot precision-recall (test sample), it looks horrible. Kind of like the ...
4
votes
4answers
151 views

Log loss vs accuracy for deciding between different learning rates?

While model tuning using cross validation and grid search I was plotting the graph of different learning rate against log loss and accuracy separately. Log loss When I used log loss as score in ...
1
vote
0answers
41 views

Decent ROC, but horrible Precision-Recall curve

I was working on a model with following process: Split to training/validation/test sets Try a series of different models like GBM, RF, Logistic Regressions Optimize hyper-params on them using ...
0
votes
1answer
988 views

Semantic segmentation: mean IOU in presence of missing classes

It seems to me that the mean IOU is a poor metric in the presence of unbalanced classes. E.g., suppose I have 10 classes but one image has only 2 classes present in its label. Consider the prediction ...
2
votes
0answers
54 views

Why is the area under the precision-recall curve not used as scoring function more often?

When I am training and evaluating classifiers or hyperparameter tuning I don't like to look at precision and recall metrics alone because those numbers depend on a threshold that I will set afterwards ...
2
votes
1answer
68 views

What's the best metric for evaluate an estimator for a multi class problem with class imbalance dataset?

accuracy, precision, f1, ROC are good for binary single class problem. but for more complex problem (imbalance multi-class problem), what should i use? Do you have any recommendation?
1
vote
2answers
138 views

Scikitlearn grid search random forest using oob as metric?

Have looked at data on oob but would like to use it as a metric in a grid search on a Random Forest classifier (multiclass) but doesn't seem to be a recognised scorer for the scoring parameter . I do ...
1
vote
0answers
140 views

How to compute G-mean score?

I would greatly appreciate if you could let me know how to fix the following issue: I used sklearn.metrics.fowlkes_mallows_score to compute G-mean score for my binary classification problem, but it ...
2
votes
0answers
23 views

What would be the main and essential criteria for evaluating auto-sklearn library ?

I m running experiments using benchmark datasets with auto-sklearn to see how its performance is different to the standard sklearn library, Since automl does an exhaustive search over parameters and ...
8
votes
2answers
347 views

Why is the F-measure preferred for classification tasks?

Why is the F-measure usually used for (supervised) classification tasks, whereas the G-measure (or Fowlkes–Mallows index) is generally used for (unsupervised) clustering tasks? The F-measure is the ...
0
votes
1answer
94 views

What is the correct way to compute lift in lift charts

How is "lift" computed? i was reading about "Gain and lift charts" in data science. I picked the following example from https://www.listendata.com/2014/08/excel-template-gain-and-lift-charts.html ...
1
vote
1answer
110 views

can accuracy rise while precision and recall drop?

I am working on a model and running some experiments, I see that under some configurations, The accuracy rises while the recall and precision are much lower, what is the mathematical explanation? is ...
0
votes
1answer
19 views

Metric to combine average with number of accurance

I was analysing a dataset which has got two main columns which I can name like: category:String amount: Int So I wanted to know the average of amount for each of the category. So I did something ...
0
votes
1answer
24 views

Does recall has different interpretation when comes to classification and information retrieval

Recall Definition In terms of classification The recall is defined as no of positive instances that are correctly detected by the classifier. $$ TP = \frac{TP}{(TP+FN)} $$ In terms of ...
1
vote
1answer
46 views

How to interpret PR and ROC Curve for an unbalanced test set

I have trained a neural network on a dataset, the test set is very unbalanced, ratio between positive examples and negatives is 1:25000. All positive examples are correctly predicted, instead ...
3
votes
0answers
918 views

Precision recall loss function

I've been using precision and recall as my metrics, as per keras-team/keras/pull/9393/files Sensitivity & specificity is what I want to optimise for. Every epoch I output it: ...
2
votes
0answers
19 views

Classification problem: custom minimization measure

Assume a binary classification problem, with $1$ denoted as a "bad" outcome, and $0$ as a "good" outcome. If it's relevant, in the sample there are significantly more bads than goods. I'm trying to ...