Questions tagged [metric]
A metric is a way to evaluate the performance of a machine learning model. Depending on the task, different metrics may be used.
249
questions
1
vote
1
answer
23
views
Is it bad to average several MAEs calculated from chunks of a big test dataset?
In my regression problem, I am using Mean Absolute Error (MAE) as a metric for my network. My test dataset is too big to fit in memory, so I am reading the test dataset in chunks and then Keras' ...
- 205
0
votes
0
answers
10
views
Survival analysis metric on time series data
I created a model that estimates the probability of failure of an asset (based on Weibull CDF, value between 0 and 1). I have a data point every minute.
I want to measure the model's success based on ...
- 1
0
votes
1
answer
31
views
Why is the sprase categorical accuracy decreasing every epoch and predictions are always NaN?
Problem Summary
My model is built and compiled properly but gets the NaN validation loss on all epochs. The training set accuracy is also infinitesimally small and keeps decreasing. I couldn't find a ...
- 113
0
votes
0
answers
49
views
Custom loss and metric functions including additional parameter in Keras
The following example is based on this approach.
Similar to that approach, I am wanting to pass an additional parameter with y_true for my custom metric, as both will be used in the computation of ...
- 1
0
votes
1
answer
23
views
When is Recall@k useful for a classifier with softmax-like output?
If a 3-class classifier returns a length-3 vector of probabilities, e.g. [0.1, 0.85, 0.05] for classes A, B, and C respectively (strongly indicating B), does it ...
- 103
0
votes
1
answer
31
views
Different accuracy scores with sklearn roc_auc_score on same model using sklearn.metrics
Why do these below lines give different outputs while the input is the same?
I need to report these results in paper, but I am unsure which is better and why.
...
- 101
0
votes
0
answers
16
views
Expected Calibration Error vs Cross-Entropy Loss
When would I use Expected Calibration Error over Cross Entropy Loss. I think I understand when to use Cross Entropy Loss i.e. it is easy to optimize Cross Entropy Loss during training. But can I not ...
- 187
1
vote
0
answers
47
views
Is there an elegant way to quantify the mix shift effect?
Imagine there is an online auction business selling 1000+ varieties of fruits. The demand for these fruits change over time, and the prices of these fruits is set by the seller.
Supposed I am ...
- 111
0
votes
1
answer
82
views
Best metrics to evaluate the performance of a regression model?
I've just started with machine learning and I have a lot to learn but one of the recent problems I'm facing is evaluating the performance of a regression model. I know about MSE, RMSE, MAE ...
2
votes
1
answer
95
views
Implementation of spBLEU
I was looking for a way to explore evaluation metrics for language translation models and I came across spBLEU. I can’t find any implementations/examples that would help me start. Does anyone have a ...
- 33
0
votes
0
answers
12
views
Does scikit learns implementation of silhouette score support parallelization and will benefit from multiple CPUs?
I wish to use the silhuette score to get the optimum number of clusters. I know kmeans implementation in scikit learn supports parallelization. But I am unsure whether the same is true for silhouette ...
0
votes
1
answer
65
views
Standard metric for distance between two clusters
Let $A=\{A_1,A_2,\cdots,A_m\}$ and $B=\{B_1,B_2,\cdots,B_n\}$ be two sets of points in $k$-dimensional Euclidean space. Each points $A_i$ or $B_i$ can be thought of as a feature vector of a data ...
- 1
0
votes
1
answer
91
views
MPE (Most Probable Explanation) vs. MAP (Maximum A Posteriori)
What is MPE?
How do MPE and MAP differ? Any example of when they would produce different results?
0
votes
0
answers
58
views
Binary classification metrics for one-hot label encoding in Tensorflow
I run a binary classification using different CNN versions in Tensorflow.
When I label samples from each class using 0 and 1, I select a sigmoid output in the last layer of the CNN, like
...
- 101
0
votes
0
answers
31
views
Name of metric: percentage of K for 100% recall
I have a recommendation system problem where full recall is important. Thus, the standard metrics of recall@k is insufficient. Rather, what I want to measure is how much of the recommendations must be ...
- 101
0
votes
1
answer
19
views
Best measure to inform how predicted value can differ from real one
I have trained a regression model and obtained a pandas series of the predicted values.
I am working on a "calculator" that will be able to return a predicted value after entering an input ...
- 15
2
votes
2
answers
490
views
Some simple questions about confusion matrix and metrics in general
I will first tell you about the context then ask my questions.
The model detects hate speech and the training and testing datasets are imbalanced (NLP).
My questions:
Is this considered a good model?
...
- 89
0
votes
0
answers
19
views
torchmetrics BinaryMatthewsCorrCoef outputs 0 if target and prediction contains only one case either positive or negative case
I stared using MCC(Matthew's correlation coefficient) metric. But getting unexpected values when the given target and pred contains only one case either positive or negative (case - 1), The output of ...
- 1
4
votes
3
answers
363
views
Which metric to use for imbalanced data in TensorFlow/Keras
I am doing a binary classification task with Keras and my model directly outputs either 0 or 1. Typically I compile the model like something below:
...
- 153
0
votes
1
answer
667
views
How to correctly measure the inference time and FLOPs of a model?
For some reason, I can’t find built-in solutions (not really?) in keras and tensorflow, while on the site
https://keras.io/api/applications/ they provide Time (ms) per inference step (CPU), but for ...
1
vote
1
answer
110
views
Questions on reproducibility of TimeGAN results
I am playing the timeGAN model, using the example code from ydata-synthetic repo.
To train the model, we used
synth.train(stock_data, train_steps=50000)
to ...
- 157
0
votes
0
answers
17
views
Finding Look alike customers
I have a set of customers and their attributes (say spend/balance/etc.) per month.
At month X something happen for some customers, but not all (call this set A).
So, now I want to find customers who ...
- 1,973
0
votes
0
answers
26
views
SemEval-2016 Task 5 Subtask 2 Evaluation
Hi, I read SEMEVAL task 5 from this website (https://alt.qcri.org/semeval2016/task5/) but the evaluation method for subtask 2 is not clear. I really appreciate it if you tell me the method. This is ...
0
votes
0
answers
77
views
Performance metrics for outlier/anomaly detection
I am currently seeking for metrics i can use to evaluate a model for outlier/anomaly detection without ground truth. The only thing i came up with for now is to use the scores/probas returned by my ...
- 1
0
votes
0
answers
34
views
Do we have any range of values for MAP@K, which are considered as good to evaluate a recommendation system
I am using MAP@K as metric for the recommendation system I am building. Currently, I am seeing a value of 0.16, which I am not sure if can be considered as a good value. Any suggestions on what range ...
- 1
1
vote
1
answer
42
views
Order of preproccesing, avoiding leakage and metrics
I have a dataset with ~40k records and 16 columns (including the target) and I want to understand the correct process behind whole data science proccess.
This is what I did:
Performed an EDA which ...
- 11
1
vote
1
answer
70
views
0
votes
0
answers
27
views
Reporting and/or evaluation of metrics in deep learning
While I was trying to write a custom training script, I encountered the following doubt.
I see that the loss is evaluated at the end of every forward pass (i.e., a step or with a particular batch of ...
0
votes
0
answers
8
views
help with comparison between the prediction of a bayesian neural network and an analytical model
i am in a weird situation where i have a bayesian neural network used for regression and a polynomial model $f(a,b,c,d) that depends on 4 parameters and that is fitted through monte carlo methods.
i ...
- 111
0
votes
0
answers
11
views
Which F-metric to use in scientific paper?
Often, academic papers refer to F1-score, but don't explain which one they use (macro, micro, weighted). What is the common F-metric in academia?
- 101
0
votes
1
answer
29
views
How to aggregate the metrics from two different regression problems?
I'm about to conduct some tests to compare two solutions to regression problems. And to make the results more robust, I want to apply both on a few different datasets (all problems will be a ...
- 277
0
votes
0
answers
14
views
Why are decision trees driven by the Gini impurity as opposed to the accuracy? [duplicate]
It seems that most implementations of decision trees use the Gini impurity as their partitioning criterion. Why isn't accuracy used instead, since it's a more widespread metric across different ...
- 203
0
votes
0
answers
69
views
How to calculate the gradient with triplet loss learning?
I have a CNN (convolutional neural network) that I train for face recognition. When teaching, I choose 3 images: Anchor, Positive, Negative. I pass each of them through CNN. Then I calculate the ...
0
votes
0
answers
7
views
Ages deviation in custom metric
I'm making custom metric for evaluation of genetic algorithm made to form teams of most appropriate people. I.e. I'd wanted to be people from one city, similar <...
- 137
1
vote
1
answer
247
views
YOLO : why does changing the confidence threshold change the [email protected]?
I trained a YOLOv7 model for a detection task. I have only one class, which is the object I want to detect. I ran test.py with --conf-thresh to 0.001 (default) and a second time with --conf-thresh to ...
- 11
0
votes
1
answer
34
views
Get dependant probabilities in multiclassification
After training my CatBoostClassifier model I call get_proba function which returns me list of probabilities. The problem starts from an another point... I transfer that data into dataframe then to ...
1
vote
2
answers
67
views
How to evaluate Natural Question-Answer Generation pairs?
I am trying to generate Natural Question-Answer for a specific domain. I am using a Large Language Model (LLM). I have only context to generate question-answers but don't have any ground truth. How to ...
- 415
2
votes
1
answer
277
views
Why is accuracy not a useful measure for information retrieval problems?
I have been studying about information retrieval and recommender systems. While reading about it I found that accuracy not a useful measure in information retrieval.
I understand that, accuracy might ...
- 31
0
votes
0
answers
61
views
When using custom metrics, num_threads setting is not working in lightgbm?
When I train a lightgbm model with my own custom metrics, I find that model is trained in single thread, though I set "num_threads": 16.
...
- 21
0
votes
1
answer
35
views
Success metric of database migration using row counts
Description
I have a problem where I'm tasked to successfully transform and repurpose data from one SQL server to another. Call the source $\text{src}$ and the target database $\text{tgt}$.
In order ...
- 101
2
votes
1
answer
3k
views
Accuracy vs Categorical Accuracy
I was running a DNN model that uses ResNet50 for Transfer Learning. While fitting the training data on my model to check the initial trend (would run for more epochs if initial trend seems right), I ...
- 77
3
votes
1
answer
56
views
What does precision-recall curve and ROC curve tell us abouth threshold invariance
Consider a binary classification problem.
Intuitively, a value for the area under the curve (for both curves) very close to 1, shows that the curve is almost L-shaped.
Thus, this means that the value ...
- 636
-1
votes
2
answers
166
views
Classification model- which metric to choose to evaluate
If my project is "Music Genre Classification"
Which metric method I need to choose to evaluate, and why ?
Thanks a lot
- 9
0
votes
1
answer
100
views
About the Evaluation method of the Market 1501 ReID dataset
The market 1501 dataset has train, query and gallery folders, each containing multiple views of people from multiple cameras. I would like to understand how to evaluate a model (trained with triplet ...
- 101
0
votes
0
answers
45
views
My overfitted decision tree regressor gives better result than pre-pruned tree?
I create a decision tree regressor without giving any parameters. The resulting tree has 6255 leaf nodes (out of 6348 entries of train set) and depth of 39. Most probably it has overfitted. But its ...
- 1
1
vote
0
answers
85
views
Calculationg perplexity (in natural language processing) manually
I am trying to understand Perplexity within Natural Language Processing as a metric more fully. And I am doing so by creating manual examples to understand all the component parts. Is the following ...
- 125
0
votes
1
answer
16
views
Statistic methods to compute the average of a list of metric results
I have a Machine Learning model that fits and predicts many time series at once, so, for each time series I have a metric result, for example, MAE. What I need is to generate an unique value for that ...
- 101
0
votes
0
answers
40
views
Am I calculating LogAUC / pAUC correctly?
Hope you are well.
I was wondering how one would calculate logAUC? I have an implementation but I don't think it's correct.
I'm trying to recreate the metric in this manuscript. See figure 2.
Any help ...
2
votes
1
answer
144
views
What makes an ROC curve a curve and why do the values change?
I have a problem. I am currently looking at a classifier and I would like to examine this using an ROC curve as a metric. However, questions have arisen to which I can not find an answer.
A ROC curve ...
- 89
0
votes
1
answer
81
views
Why compare multiple machine learning algorithms and then decide which algorithm to use for fine tuning?
I have a problem.
There is a dataset A, which deals with a classification problem. And for this dataset, several different baseline algorithms have been defined and computed.
In addition, three models ...
- 89