Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Evaluating accuracy of TensorFlow object detection model

I've just finished training and testing my TensorFlow object detection model, and would like to create a presentation displaying various statistics that demonstrate the overall accuracy and utility of ...
1
vote
0answers
12 views

What's the difference between these two custom sparse categorical accuracy functions?

I have a sequence classification model featuring CustomELMo Embeddings layer + BiLSTM + Fully Connected layer. I've found two custom metrics for sparse_categorical_accuracy, but can't wrap my head ...
2
votes
1answer
16 views

How can I get an algorithm to have an evalutation metric based on aggregate predictions?

Let's say I have a model that makes a prediction per individual. An example data set is below. Normally, evaluation metrics (for example within the XGBoost algorthim), are used at the individual ...
1
vote
0answers
14 views

How does the feval parameter influences the XGBoost training process?

In the package XGBoost, is possible to modify the feval (evaluated function) to a personalized one (as shown in the link: MAPE eval metric). I would like to know how is the training process of the ...
2
votes
1answer
46 views

Smart data split (train/eval) for Object Detection

I am looking for a smart way of splitting object detection data (images with labelled objects inside them) while taking into account the distribution of the objects themselves and not just the images. ...
2
votes
3answers
192 views

In k-fold-cross-validation, why do we compute the mean of the metric of each fold

In k-fold-cross-validation, the "correct" scheme seem to compute the metric (say the accuracy) for each fold, and then return the mean as the final metric. Source : https://scikit-learn.org/stable/...
2
votes
1answer
21 views

Metrics for Name Entity Recognition

Working on a NER project, I have been facing the problem of evaluating my model during training. I cannot be using the accuracy metrics or f1 score or any other metrics to evaluate my model on runtime ...
0
votes
0answers
17 views

Which metric should be used to select best binary pixel-wise classifier for segmentation task?

I am doing a semantic segmentation task using a supervised algorithm to classify image pixels into one class or the other (binary classification). I am trying several classifiers and feature ...
0
votes
1answer
28 views

A robust metric in the presence of class imbalance

When evaluating the performance of a multiclass classification problem, on a highly imbalanced dataset, what is the most robust metric for this purpose? I read a paper that states: "Average ...
0
votes
1answer
39 views

Normalized metric for comparing regression models performance

I was recently trying to explain to someone whether performance of my estimation approach is good or bad. For instance, whether a model with Mean Absolute Error (MAE) of 17000 is a bad solution. It ...
0
votes
0answers
11 views

PageRank computation with dumping (scaling) factor

it is not clear to me how to calculate two iterations of PageRank computation on the following network with dumping (scaling) factor s = 1.how can they calculate it correctly?
0
votes
1answer
14 views

Evaluating performance of classifier on lopsided dataset

I have a binary classifier that I would like to evaluate the performance of. It's been both trained and tested on a data set where the ratio of true to false labels is lopsided. This means that while ...
0
votes
0answers
16 views

How can I evaluate my sequence prediction model?

I want to evaluate the performance of my prediction model , which is an VED (Variational Encoder Decoder) used for sequences prediction (it predicts the next sequence knowing the actual) I want to ...
1
vote
2answers
62 views

Do precision-recall curves have a constant shape/pattern?

I know ROC curve always looks like a stair shape and that I can evaluate AUC of ROC. And I know I can compute AUC of ROC curve to compare which model is better. What I wonder is: Does precision-...
4
votes
2answers
85 views

Is the F1 Score sensitive to the threshold?

Is the F1 score sensitive or indifferent to the threshold (for defining positive or negative)?
0
votes
0answers
13 views

Error-analysis and evaluation of a model using Python?

My method of evaluating a model is the following : Split the training data set and do cross validation to obtain an accuracy of my model on my cross validation data set. Use the parameters that gave ...
1
vote
1answer
32 views

Logistic Regression - ROC curve plots Sensitivity vs Specificity instead of (1-Specificity)

I am new to Machine Learning and have been doing some practice on Logistic Regression. To evaluate the models, I've been trying to create some ROC plots. The package that i used is pROC. The model ...
0
votes
0answers
13 views

Original References for Micro and Macro Averages

When evaluating group results, micro and macro averages are commonly used. They are explained in multiple text books in detail. However, I wonder whether there is an "original" inventor to those ...
1
vote
1answer
21 views

How is the linear regression cost function evolved?

A couple of weeks ago I joined the Standford University machine learning course on Coursera. In that course, they directly gave the cost function formula without telling how this formula was evolved. ...
0
votes
0answers
34 views

Dummy/baseline models for time series forecasting

I am working on an evaluation of time series forecasting models in Python, more specifically with statsmodels, scikit-learn and <...
0
votes
1answer
41 views

Metric (rather than RMSE, MSE, etc.) to choose the best model in terms of the ability to detect peaks better

I have created multiple regression models and wanted to choose the best one. One common metric would be RMSE, as you know. When I looked at the results, second model (RMSE = 0.15) was better able to ...
2
votes
1answer
25 views

Conditional Entropy and Mutual Information - Clustering evaluation

First of all, I am doing clustering and I have the true labels for my data. For evaluation, I am using the weighted average of the entropy values for each predicted cluster. I also came across with ...
0
votes
0answers
53 views

Can McNemar's test be applied to evaluate multiclass models?

Full Disclosure: I did a semi-cross post of this question due to low traffic on Cross Validated. Once I get an answer on any of the two questions, I will link the answer back to the respective other. ...
2
votes
0answers
39 views

Is AUC a good metric for evaluating the performance of a multi-class classification?

Considering the definition of AUC (Area Under Curve), is that a reliable performance metric for a multi-class (30-40 classes) classification problem?
1
vote
2answers
48 views

Classifier performance evaluation

I have an unbalanced dataset which has 920 samples in total, 689 belong to the first class, and 222 to second class. and both classes are significant for me. so when building a classifier model such ...
5
votes
1answer
121 views

How to compare two unsupervised anomaly detection algorithms on the same data-set?

I want to solve an anomaly detection problem on an unlabeled data-set. The only information about this problem is that the anomalies population is lower than 0.1%. It should be notice that the size of ...
2
votes
0answers
20 views

Obtain learning curve of Gradient Boosted Tree model in PySpark

Currently there seems to be no method in PySpark of checkpointing the performance of a model at each gradient update. Is there a way to get the performance of a model at each gradient update so that a ...
1
vote
0answers
57 views

What does the ratio of precision/recall to their ideal values mean, and why are they equal?

When evaluating rankings, Normalized Discounted Cumulative Gain (NDCG) normalizes the score to a [0,1] range by dividing by the ideal score. What happens if we take the same idea to Precision (Pre) ...
2
votes
1answer
785 views

Micro-F1 and Macro-F1 are equal in binary classification and I don't know why

I have a binary classification problem which in the test set, the number of data in both classes are equal (the test number of class 0 and class 1 are equal). Since we know that the number of samples ...
1
vote
1answer
213 views

Can macro F1 score be greater than micro F1 score?

I am reading about evaluation metrics, and it seems that micro scores are more useful. But I was wondering about scenarios where macro F1 score is greater than micro F1 score, and if this is at all ...
5
votes
2answers
38 views

Does a precision score increasing with a higher number of folds mean the model will improve with more data?

I have been working on a pretty simple text classifying module (tfidf + Random Forest). My manager insisted on using a simple .7/.3 split rather than doing cross validation, then was adamant about ...
2
votes
1answer
44 views

Ranking ATM based on Utilization and Economic Data (Scoring/Rank Model)

I have a sample data of around 10 ATM's Locations along with their Utilization Count (Deposits, Withdrawals and Others) for the past 3 months. I am planning to collect additional data such as nearby ...
0
votes
1answer
36 views

Purpose of test data in binary classification

I have a highly biased training dataset where AppId 6,7,8,9,10 are almost never purchased. I made this up just to see how good my comprehension of calculating the classification metrics is such as <...
0
votes
0answers
20 views

multi armed bandit offline evaluation

I have read those papers: Counterfactual Estimation and Optimization of Click Metrics for Search Engines Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation ...
1
vote
1answer
285 views

Evaluation of semantic segmentation network with mAP

I am interested in evaluating a semantic segmentation network. I've seen lots of challenges such as PASCAL VOC use the mean average precision metric(mAP). I understand how this would work with an ...
0
votes
1answer
959 views

How Do I Find the Matthews Correlation Coefficient (MCC) in Python? [closed]

I would like to find the Matthews Correlation Coefficient (MCC) for predictions made by a binary Sklearn machine learning model. How do I do it?
1
vote
2answers
93 views

Doubt to use accuracy or macro f1 measure in an unbalanced classification task

I have a multi-class classification task where the organizers said that the final results will be using the Accuracy measure. The provided data is unbalanced, and I don't have an idea about the test ...
1
vote
0answers
107 views

NDCG score is greater than 1

I'm solving a problem of ranking classes for each unique id based on the utilization quantity. I have 6 unique classes in the training and test data. My neural net mode predicts the utilization ...
1
vote
2answers
449 views

Both train and test error are decreasing in XGBoost iterations

I have an issue with training an XGBoost classifier in a sence that both train and test error only decrease throughout more iterations (num_boost_round) even if I use 1000 num boost rounds and 10 ...
1
vote
1answer
36 views

Evaluation of linear regression model

I want to evaluate the performance of my linear regression model. I have the true values of y (y-true). I am thinking of two way for evaluation but not sure which one is correct. Let's assume that ...
1
vote
0answers
20 views

What does AR(max=1) mean?

http://cocodataset.org/#detection-eval "ARmax=1: AR given 1 detection per image" "AR is the maximum recall given a fixed number of detections per image, averaged over categories and IoUs." Above link,...
1
vote
0answers
13 views

Validating performance of panel data based models

I'm wondering from a theoretical/general practice perspective, what is the best way to evaluate performance of regression models derived from panel data (i.e. a time series of cross sectional data). ...
3
votes
3answers
67 views

Statistical test for machine learning

I want to prove that my proposed machine learning algorithm (prop_ml) is better than other baseline algorithms (ml_1, ml_2, ml_3) when given a small number of data for training. What I've done is to ...
1
vote
0answers
54 views

very large difference between cross_val and (multiple) r2 model evaluation

I did submit my first kaggle kernel, on the avocado dataset kernel link, I treated it like I should predict the avocado price so I splitted the dataset in a train & test set, fitted the model and ...
4
votes
0answers
186 views

Scikit-learn average_precision_score() vs. auc score of precision_recall_curve()

I've been searching around for an explanation to this, and haven't come across one yet- in scikit-learn, when I compute the auc() of the ...
2
votes
0answers
24 views

What would be the main and essential criteria for evaluating auto-sklearn library ?

I m running experiments using benchmark datasets with auto-sklearn to see how its performance is different to the standard sklearn library, Since automl does an exhaustive search over parameters and ...
5
votes
2answers
2k views

Macro- or micro-average for imbalanced class problems

The question of whether to use macro- or micro-averages when the data is imbalanced comes up all the time. Some googling shows that many bloggers tend to say that micro-average is the preferred way ...
8
votes
2answers
382 views

Why is the F-measure preferred for classification tasks?

Why is the F-measure usually used for (supervised) classification tasks, whereas the G-measure (or Fowlkes–Mallows index) is generally used for (unsupervised) clustering tasks? The F-measure is the ...
2
votes
1answer
96 views

Chi-square as evaluation metrics for nonlinear machine learning regression models

I am using machine learning models to predict an ordinal variable (values: 1,2,3,4, and 5) using 7 different features. I posed this as a regression problem, so the final outputs of a model are ...
0
votes
1answer
164 views

Evaluate prediction from multiple classification model

Given that I have data containing images of oranges, apples and pineapples and I want to classify depending on a set of features. Expected that I have completed the model and ready for prediction. ...