Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

0
votes
0answers
6 views

Can macro F1 score be greater than micro F1 score?

I am reading about evaluation metrics, and it seems that micro scores are more useful. But I was wondering about scenarios where macro F1 score is greater than micro F1 score, and if this is at all ...
4
votes
2answers
20 views

Does a precision score increasing with a higher number of folds mean the model will improve with more data?

I have been working on a pretty simple text classifying module (tfidf + Random Forest). My manager insisted on using a simple .7/.3 split rather than doing cross validation, then was adamant about ...
2
votes
1answer
27 views

Ranking ATM based on Utilization and Economic Data (Scoring/Rank Model)

I have a sample data of around 10 ATM's Locations along with their Utilization Count (Deposits, Withdrawals and Others) for the past 3 months. I am planning to collect additional data such as nearby ...
0
votes
1answer
29 views

Purpose of test data in binary classification

I have a highly biased training dataset where AppId 6,7,8,9,10 are almost never purchased. I made this up just to see how good my comprehension of calculating the classification metrics is such as <...
0
votes
0answers
6 views

multi armed bandit offline evaluation

I have read those papers: Counterfactual Estimation and Optimization of Click Metrics for Search Engines Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation ...
0
votes
1answer
31 views

Evaluation of semantic segmentation network with mAP

I am interested in evaluating a semantic segmentation network. I've seen lots of challenges such as PASCAL VOC use the mean average precision metric(mAP). I understand how this would work with an ...
0
votes
0answers
14 views

Why AIC is being used as a validating criteria for survival models?

Recently our team got a use case to create a model that estimates failures among machines. While going through survival analysis papers and articles, we found that the validating measure is always AIC....
0
votes
1answer
157 views

How Do I Find the Matthews Correlation Coefficient (MCC) in Python? [closed]

I would like to find the Matthews Correlation Coefficient (MCC) for predictions made by a binary Sklearn machine learning model. How do I do it?
0
votes
0answers
17 views

Comparing sequences

Let's say that I have an expected sequence that look like this : - - - - - A - - - - B - - C - D - - - A A A B A - - - C - - B - - - - - - And I have two ...
1
vote
0answers
39 views

NDCG score is greater than 1

I'm solving a problem of ranking classes for each unique id based on the utilization quantity. I have 6 unique classes in the training and test data. My neural net mode predicts the utilization ...
1
vote
2answers
92 views

Both train and test error are decreasing in XGBoost iterations

I have an issue with training an XGBoost classifier in a sence that both train and test error only decrease throughout more iterations (num_boost_round) even if I use 1000 num boost rounds and 10 ...
1
vote
1answer
26 views

Evaluation of linear regression model

I want to evaluate the performance of my linear regression model. I have the true values of y (y-true). I am thinking of two way for evaluation but not sure which one is correct. Let's assume that ...
1
vote
0answers
17 views

What does AR(max=1) mean?

http://cocodataset.org/#detection-eval "ARmax=1: AR given 1 detection per image" "AR is the maximum recall given a fixed number of detections per image, averaged over categories and IoUs." Above link,...
1
vote
0answers
8 views

Validating performance of panel data based models

I'm wondering from a theoretical/general practice perspective, what is the best way to evaluate performance of regression models derived from panel data (i.e. a time series of cross sectional data). ...
2
votes
3answers
61 views

Statistical test for machine learning

I want to prove that my proposed machine learning algorithm (prop_ml) is better than other baseline algorithms (ml_1, ml_2, ml_3) when given a small number of data for training. What I've done is to ...
1
vote
0answers
34 views

very large difference between cross_val and (multiple) r2 model evaluation

I did submit my first kaggle kernel, on the avocado dataset kernel link, I treated it like I should predict the avocado price so I splitted the dataset in a train & test set, fitted the model and ...
0
votes
0answers
14 views

How to set score for a set of evaluations based on the number of responses?

Our company has an survey module where our clients evaluate our employees. Each employee can have from 1 to N evaluations being N equal to the total number of clients. The evaluation has several ...
3
votes
0answers
111 views

Scikit-learn average_precision_score() vs. auc score of precision_recall_curve()

I've been searching around for an explanation to this, and haven't come across one yet- in scikit-learn, when I compute the auc() of the ...
2
votes
0answers
18 views

What would be the main and essential criteria for evaluating auto-sklearn library ?

I m running experiments using benchmark datasets with auto-sklearn to see how its performance is different to the standard sklearn library, Since automl does an exhaustive search over parameters and ...
2
votes
1answer
495 views

Macro- or micro-average for imbalanced class problems

The question of whether to use macro- or micro-averages when the data is imbalanced comes up all the time. Some googling shows that many bloggers tend to say that micro-average is the preferred way ...
6
votes
1answer
172 views

Why is the F-measure preferred for classification tasks?

Why is the F-measure usually used for (supervised) classification tasks, whereas the G-measure (or Fowlkes–Mallows index) is generally used for (unsupervised) clustering tasks? The F-measure is the ...
0
votes
0answers
27 views

AVOD Object Detection Library Using Point Cloud and Image Data

I downloaded and used the AVOD object detection library using the code and instructions available on the following github link. https://github.com/kujason/avod I ...
2
votes
1answer
58 views

Chi-square as evaluation metrics for nonlinear machine learning regression models

I am using machine learning models to predict an ordinal variable (values: 1,2,3,4, and 5) using 7 different features. I posed this as a regression problem, so the final outputs of a model are ...
0
votes
1answer
83 views

Evaluate prediction from multiple classification model

Given that I have data containing images of oranges, apples and pineapples and I want to classify depending on a set of features. Expected that I have completed the model and ready for prediction. ...
1
vote
0answers
72 views

Evaluation of regression models with different evaluations (MSE, variance, VAF etc.)

When comparing several regression models in terms of quality, it seems like most have agreed on the MSE. There are also papers comparing "variance" and "variance accounted for (VAF)". However, there ...
3
votes
1answer
55 views

Evaluating the result of topic modeling in a way that time matters

I have run different topic modeling approach on my data(its clinical data related to Cognitive impairment diseases. we are going to process what thing is important that make it develop to more harsh ...
0
votes
0answers
11 views

Partial dependence plot vs prediction of an averaged row

I have been trying to predict property prices and I want to produce a lat, long partial dependence plot to visualize how much the location of a property affect the price prediction. The formal way of ...
1
vote
1answer
33 views

How to interpret PR and ROC Curve for an unbalanced test set

I have trained a neural network on a dataset, the test set is very unbalanced, ratio between positive examples and negatives is 1:25000. All positive examples are correctly predicted, instead ...
1
vote
1answer
29 views

Clustering evaluation metrics with subquadratic time complexity

It exists many evaluation metrics but often they are quadratic or more on number of data points preventing any application on massive data sets as RAND or Silhouette indexes. For the moment i used : ...
0
votes
0answers
23 views

How to measure model improvement for Recommender Systems in real applications?

In the academia, model 'goodness' for recommendation systems are typically in terms of a loss or metric (i.e. MSE loss, Mean Average Precision). In real world applications, companies would deploy A/B ...
6
votes
3answers
4k views

What is the difference between bootstrapping and cross-validation?

I used to apply K-fold cross-validation for robust evaluation of my machine learning models. But I'm aware of the existence of the bootstrapping method for this purpose as well. However, I cannot see ...
0
votes
1answer
375 views

Calculate average Intersection over Union

I want to have a global IoU metric for each class in a segmentation model with a neural net. The idea is, once the net is trained, doing the forward pass over all training examples an calculate the ...
3
votes
3answers
307 views

Evaluation methods for multi-class classification

I am looking for single-number evaluation method that can be used in multi-class classification tasks that take into account imbalanced data-sets. For instance, ...
8
votes
4answers
6k views

When is precision more important over recall?

Can anyone give me some examples where precision is important and some examples where recall is important ?
2
votes
0answers
153 views

Hyperparameter tuning in multiclass classification problem: which scoring metric?

I'm working with an imbalanced multi class dataset. I try to tune the parameters of a DecisionTreeClassifier, ...
2
votes
2answers
57 views

Size of folds in k-fold cross-validation

When evaluating results using cross-validation, several strategies can be adopted, as using 5 or 10 folds, or doing leave one out cross-validation, as well as doing a 80/20 split. Under which are ...
4
votes
1answer
132 views

Class leaking on validation set

I am quite new in the ML field. I think I correctly understood the information leaking problem during the testing/validation phases but I am struggling to understand some François Chollet statements ...
1
vote
1answer
178 views

Is recall more important than precision for mass mailings?

Say for example, I built a classification model for a mailing campaign that will be applied to 1M records. The positive class for the model would be customers and the negative records would be non-...
0
votes
1answer
422 views

Python - Calculate Cost profitability and benefit of the model

I've this code in Python in order to calculate the precision of my model and to print confusion matrix using Decision Trees Classifier: ...
1
vote
1answer
166 views

What is the advantage of using Dunn index over other metrics for evaluating clustering algorithm? [closed]

There are many metrics to evaluate clustering algorithm like Calinski-Harabaz Index, Dunn index, Rand index, etc. Are there any advantage of using Dunn index over other metrics for evaluating ...
1
vote
0answers
62 views

Performance Metric for topic extraction when there is no ground truth

I am extracting topics from text using a predefined ontology containing 2690 concepts, wordnet(to expand concept terms with their synsets, and other morphological forms of the same word) and lucene to ...
2
votes
1answer
20 views

Find threshold in rate to determine reason for lost customer

I'm not sure if anybody will be able to help with this, or even if I'll be able to explain it well but I am stuck so here goes.... I have a set of customers, some are lost, some are still active. We ...
4
votes
3answers
197 views

Evaluate new features

How should I evaluate whether new features are effective or not? Should I build a new model with the new features then compare with the old one with the same hyper parameter?
0
votes
1answer
577 views

How to measure F1 score and NMI for clustering task?

I see the authors of this paper are measuring F1 and NMI scores to measure the clustering quality. However, I don't understand the algorithm of how they actually measure it. See the Evaluation Section....
1
vote
0answers
17 views

Do I need to use Bayes to combine a sample's class probability with the performance of the overall model?

For a classification algorithm that gives the predicted probabilities for each class (ie random forest in sklearn), by default the classes are separated with a score of 0.5. After evaluating the ...
0
votes
1answer
34 views
3
votes
1answer
1k views

Splitting hold-out sample and training sample only once?

I have a question related to evaluating out-of-sample predictions. For my research I want to tune two parameters related to Support Vector Machines, and use these optimized parameters to predict the ...
0
votes
1answer
146 views

Clustering documents - how to evaluate results?

I'm using DBSCAN clustering on a set of documents. The documents' content was converted to TF-IDF matrix, and I'd like to find consistent ways to evaluate the clusters when no added information is ...
1
vote
1answer
104 views

How to evaluate sequence to sequence models?

I wonder how to evaluate variable long sequence-to-sequence predictions? Let us say I have the following $Y$ and $\hat{Y}$ $Y = [["1", "2", "2"], ["3", "2", "2"], ["1", "3", "2", "2"]]$ $\hat{Y} = [[...
4
votes
1answer
141 views

How can RL agents be monitored?

My question is about how to monitor RL agents in production. To make the question easier to discuss, here is a use case. Please don't focus on difficulties in implementing such an agent, but rather on ...