Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

0
votes
1answer
15 views

Normalized metric for comparing regression models performance

I was recently trying to explain to someone whether performance of my estimation approach is good or bad. For instance, whether a model with Mean Absolute Error (MAE) of 17000 is a bad solution. It ...
0
votes
0answers
8 views

PageRank computation with dumping (scaling) factor

it is not clear to me how to calculate two iterations of PageRank computation on the following network with dumping (scaling) factor s = 1.how can they calculate it correctly?
0
votes
1answer
10 views

Evaluating performance of classifier on lopsided dataset

I have a binary classifier which I would like to evaluate the performance of. It's both trained and tested on a data set where the ratio of true to false labels is lopsided. This means that while it's ...
0
votes
0answers
16 views

How can I evaluate my sequence prediction model?

I want to evaluate the performance of my prediction model , which is an VED (Variational Encoder Decoder) used for sequences prediction (it predicts the next sequence knowing the actual) I want to ...
1
vote
2answers
43 views

Do precision-recall curves have a constant shape/pattern?

I know ROC curve always looks like a stair shape and that I can evaluate AUC of ROC. And I know I can compute AUC of ROC curve to compare which model is better. What I wonder is: Does precision-...
4
votes
2answers
73 views

Is the F1 Score sensitive to the threshold?

Is the F1 score sensitive or indifferent to the threshold (for defining positive or negative)?
0
votes
0answers
9 views

Error-analysis and evaluation of a model using Python?

My method of evaluating a model is the following : Split the training data set and do cross validation to obtain an accuracy of my model on my cross validation data set. Use the parameters that gave ...
1
vote
1answer
30 views

Logistic Regression - ROC curve plots Sensitivity vs Specificity instead of (1-Specificity)

I am new to Machine Learning and have been doing some practice on Logistic Regression. To evaluate the models, I've been trying to create some ROC plots. The package that i used is pROC. The model ...
0
votes
0answers
11 views

Original References for Micro and Macro Averages

When evaluating group results, micro and macro averages are commonly used. They are explained in multiple text books in detail. However, I wonder whether there is an "original" inventor to those ...
1
vote
1answer
18 views

How is the linear regression cost function evolved?

A couple of weeks ago I joined the Standford University machine learning course on Coursera. In that course, they directly gave the cost function formula without telling how this formula was evolved. ...
0
votes
0answers
14 views

Dummy/baseline models for time series forecasting

I am working on an evaluation of time series forecasting models in Python, more specifically with statsmodels, scikit-learn and <...
0
votes
1answer
35 views

Metric (rather than RMSE, MSE, etc.) to choose the best model in terms of the ability to detect peaks better

I have created multiple regression models and wanted to choose the best one. One common metric would be RMSE, as you know. When I looked at the results, second model (RMSE = 0.15) was better able to ...
2
votes
1answer
24 views

Conditional Entropy and Mutual Information - Clustering evaluation

First of all, I am doing clustering and I have the true labels for my data. For evaluation, I am using the weighted average of the entropy values for each predicted cluster. I also came across with ...
0
votes
0answers
23 views

Can McNemar's test be applied to evaluate multiclass models?

Full Disclosure: I did a semi-cross post of this question due to low traffic on Cross Validated. Once I get an answer on any of the two questions, I will link the answer back to the respective other. ...
2
votes
0answers
39 views

Is AUC a good metric for evaluating the performance of a multi-class classification?

Considering the definition of AUC (Area Under Curve), is that a reliable performance metric for a multi-class (30-40 classes) classification problem?
1
vote
2answers
47 views

Classifier performance evaluation

I have an unbalanced dataset which has 920 samples in total, 689 belong to the first class, and 222 to second class. and both classes are significant for me. so when building a classifier model such ...
5
votes
1answer
99 views

How to compare two unsupervised anomaly detection algorithms on the same data-set?

I want to solve an anomaly detection problem on an unlabeled data-set. The only information about this problem is that the anomalies population is lower than 0.1%. It should be notice that the size of ...
2
votes
0answers
19 views

Obtain learning curve of Gradient Boosted Tree model in PySpark

Currently there seems to be no method in PySpark of checkpointing the performance of a model at each gradient update. Is there a way to get the performance of a model at each gradient update so that a ...
1
vote
0answers
50 views

What does the ratio of precision/recall to their ideal values mean, and why are they equal?

When evaluating rankings, Normalized Discounted Cumulative Gain (NDCG) normalizes the score to a [0,1] range by dividing by the ideal score. What happens if we take the same idea to Precision (Pre) ...
2
votes
1answer
428 views

Micro-F1 and Macro-F1 are equal in binary classification and I don't know why

I have a binary classification problem which in the test set, the number of data in both classes are equal (the test number of class 0 and class 1 are equal). Since we know that the number of samples ...
1
vote
1answer
131 views

Can macro F1 score be greater than micro F1 score?

I am reading about evaluation metrics, and it seems that micro scores are more useful. But I was wondering about scenarios where macro F1 score is greater than micro F1 score, and if this is at all ...
5
votes
2answers
35 views

Does a precision score increasing with a higher number of folds mean the model will improve with more data?

I have been working on a pretty simple text classifying module (tfidf + Random Forest). My manager insisted on using a simple .7/.3 split rather than doing cross validation, then was adamant about ...
2
votes
1answer
38 views

Ranking ATM based on Utilization and Economic Data (Scoring/Rank Model)

I have a sample data of around 10 ATM's Locations along with their Utilization Count (Deposits, Withdrawals and Others) for the past 3 months. I am planning to collect additional data such as nearby ...
0
votes
1answer
35 views

Purpose of test data in binary classification

I have a highly biased training dataset where AppId 6,7,8,9,10 are almost never purchased. I made this up just to see how good my comprehension of calculating the classification metrics is such as <...
0
votes
0answers
12 views

multi armed bandit offline evaluation

I have read those papers: Counterfactual Estimation and Optimization of Click Metrics for Search Engines Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation ...
1
vote
1answer
133 views

Evaluation of semantic segmentation network with mAP

I am interested in evaluating a semantic segmentation network. I've seen lots of challenges such as PASCAL VOC use the mean average precision metric(mAP). I understand how this would work with an ...
0
votes
1answer
707 views

How Do I Find the Matthews Correlation Coefficient (MCC) in Python? [closed]

I would like to find the Matthews Correlation Coefficient (MCC) for predictions made by a binary Sklearn machine learning model. How do I do it?
1
vote
2answers
70 views

Doubt to use accuracy or macro f1 measure in an unbalanced classification task

I have a multi-class classification task where the organizers said that the final results will be using the Accuracy measure. The provided data is unbalanced, and I don't have an idea about the test ...
1
vote
0answers
90 views

NDCG score is greater than 1

I'm solving a problem of ranking classes for each unique id based on the utilization quantity. I have 6 unique classes in the training and test data. My neural net mode predicts the utilization ...
1
vote
2answers
307 views

Both train and test error are decreasing in XGBoost iterations

I have an issue with training an XGBoost classifier in a sence that both train and test error only decrease throughout more iterations (num_boost_round) even if I use 1000 num boost rounds and 10 ...
1
vote
1answer
32 views

Evaluation of linear regression model

I want to evaluate the performance of my linear regression model. I have the true values of y (y-true). I am thinking of two way for evaluation but not sure which one is correct. Let's assume that ...
1
vote
0answers
20 views

What does AR(max=1) mean?

http://cocodataset.org/#detection-eval "ARmax=1: AR given 1 detection per image" "AR is the maximum recall given a fixed number of detections per image, averaged over categories and IoUs." Above link,...
1
vote
0answers
13 views

Validating performance of panel data based models

I'm wondering from a theoretical/general practice perspective, what is the best way to evaluate performance of regression models derived from panel data (i.e. a time series of cross sectional data). ...
2
votes
3answers
64 views

Statistical test for machine learning

I want to prove that my proposed machine learning algorithm (prop_ml) is better than other baseline algorithms (ml_1, ml_2, ml_3) when given a small number of data for training. What I've done is to ...
1
vote
0answers
44 views

very large difference between cross_val and (multiple) r2 model evaluation

I did submit my first kaggle kernel, on the avocado dataset kernel link, I treated it like I should predict the avocado price so I splitted the dataset in a train & test set, fitted the model and ...
3
votes
0answers
162 views

Scikit-learn average_precision_score() vs. auc score of precision_recall_curve()

I've been searching around for an explanation to this, and haven't come across one yet- in scikit-learn, when I compute the auc() of the ...
2
votes
0answers
22 views

What would be the main and essential criteria for evaluating auto-sklearn library ?

I m running experiments using benchmark datasets with auto-sklearn to see how its performance is different to the standard sklearn library, Since automl does an exhaustive search over parameters and ...
4
votes
2answers
1k views

Macro- or micro-average for imbalanced class problems

The question of whether to use macro- or micro-averages when the data is imbalanced comes up all the time. Some googling shows that many bloggers tend to say that micro-average is the preferred way ...
8
votes
2answers
318 views

Why is the F-measure preferred for classification tasks?

Why is the F-measure usually used for (supervised) classification tasks, whereas the G-measure (or Fowlkes–Mallows index) is generally used for (unsupervised) clustering tasks? The F-measure is the ...
2
votes
1answer
83 views

Chi-square as evaluation metrics for nonlinear machine learning regression models

I am using machine learning models to predict an ordinal variable (values: 1,2,3,4, and 5) using 7 different features. I posed this as a regression problem, so the final outputs of a model are ...
0
votes
1answer
142 views

Evaluate prediction from multiple classification model

Given that I have data containing images of oranges, apples and pineapples and I want to classify depending on a set of features. Expected that I have completed the model and ready for prediction. ...
1
vote
0answers
82 views

Evaluation of regression models with different evaluations (MSE, variance, VAF etc.)

When comparing several regression models in terms of quality, it seems like most have agreed on the MSE. There are also papers comparing "variance" and "variance accounted for (VAF)". However, there ...
3
votes
1answer
57 views

Evaluating the result of topic modeling in a way that time matters

I have run different topic modeling approach on my data(its clinical data related to Cognitive impairment diseases. we are going to process what thing is important that make it develop to more harsh ...
1
vote
1answer
42 views

How to interpret PR and ROC Curve for an unbalanced test set

I have trained a neural network on a dataset, the test set is very unbalanced, ratio between positive examples and negatives is 1:25000. All positive examples are correctly predicted, instead ...
1
vote
1answer
34 views

Clustering evaluation metrics with subquadratic time complexity

It exists many evaluation metrics but often they are quadratic or more on number of data points preventing any application on massive data sets as RAND or Silhouette indexes. For the moment i used : ...
10
votes
4answers
7k views

What is the difference between bootstrapping and cross-validation?

I used to apply K-fold cross-validation for robust evaluation of my machine learning models. But I'm aware of the existence of the bootstrapping method for this purpose as well. However, I cannot see ...
0
votes
1answer
586 views

Calculate average Intersection over Union

I want to have a global IoU metric for each class in a segmentation model with a neural net. The idea is, once the net is trained, doing the forward pass over all training examples an calculate the ...
3
votes
3answers
465 views

Evaluation methods for multi-class classification

I am looking for single-number evaluation method that can be used in multi-class classification tasks that take into account imbalanced data-sets. For instance, ...
11
votes
5answers
9k views

When is precision more important over recall?

Can anyone give me some examples where precision is important and some examples where recall is important ?
3
votes
0answers
213 views

Hyperparameter tuning in multiclass classification problem: which scoring metric?

I'm working with an imbalanced multi class dataset. I try to tune the parameters of a DecisionTreeClassifier, ...