Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [evaluation]

The tag has no usage guidance.

1
vote
0answers
12 views

NDCG score is greater than 1

I'm solving a problem of ranking classes for each unique id based on the utilization quantity. I have 6 unique classes in the training and test data. My neural net mode predicts the utilization ...
0
votes
0answers
14 views

How to evaluate accuracy for a multiclass classification algorithm which creates its own classes?

I have a classification algorithm, which (very) regularly sees data which is not part of any previously seen classes and must create a new class, which future data could be classified into. How should ...
0
votes
0answers
21 views

Address Class-Imbalance Dataset using Sampling Techniques on Train, Test Dataset or Both?

I am dealing with an unbalanced dataset and I'm really confused if I should apply sampling techniques, like downsampling, smote, upsampling etc., on the train, test dataset or both? The minority ...
1
vote
1answer
21 views

Both train and test error are decreasing in XGBoost iterations

I have an issue with training an XGBoost classifier in a sence that both train and test error only decrease throughout more iterations (num_boost_round) even if I use 1000 num boost rounds and 10 ...
1
vote
1answer
22 views

Evaluation of linear regression model

I want to evaluate the performance of my linear regression model. I have the true values of y (y-true). I am thinking of two way for evaluation but not sure which one is correct. Let's assume that ...
1
vote
0answers
14 views

What does AR(max=1) mean?

http://cocodataset.org/#detection-eval "ARmax=1: AR given 1 detection per image" "AR is the maximum recall given a fixed number of detections per image, averaged over categories and IoUs." Above link,...
1
vote
0answers
6 views

Validating performance of panel data based models

I'm wondering from a theoretical/general practice perspective, what is the best way to evaluate performance of regression models derived from panel data (i.e. a time series of cross sectional data). ...
0
votes
0answers
35 views

Metric to evaluate keras multi-class text classifiation

I saw a lot of published jupyter notebooks where a multi-class text classification was performed. In most cases there are more than 3 possible classes. However, I noticed that the models usually 'only'...
2
votes
3answers
60 views

Statistical test for machine learning

I want to prove that my proposed machine learning algorithm (prop_ml) is better than other baseline algorithms (ml_1, ml_2, ml_3) when given a small number of data for training. What I've done is to ...
1
vote
0answers
22 views

very large difference between cross_val and (multiple) r2 model evaluation

I did submit my first kaggle kernel, on the avocado dataset kernel link, I treated it like I should predict the avocado price so I splitted the dataset in a train & test set, fitted the model and ...
0
votes
0answers
14 views

How to set score for a set of evaluations based on the number of responses?

Our company has an survey module where our clients evaluate our employees. Each employee can have from 1 to N evaluations being N equal to the total number of clients. The evaluation has several ...
2
votes
0answers
59 views

Scikit-learn average_precision_score() vs. auc score of precision_recall_curve()

I've been searching around for an explanation to this, and haven't come across one yet- in scikit-learn, when I compute the auc() of the ...
2
votes
0answers
11 views

What would be the main and essential criteria for evaluating auto-sklearn library ?

I m running experiments using benchmark datasets with auto-sklearn to see how its performance is different to the standard sklearn library, Since automl does an exhaustive search over parameters and ...
2
votes
1answer
129 views

Macro- or micro-average for imbalanced class problems

The question of whether to use macro- or micro-averages when the data is imbalanced comes up all the time. Some googling shows that many bloggers tend to say that micro-average is the preferred way ...
4
votes
1answer
82 views

Why is the F-measure preferred for classification tasks?

Why is the F-measure usually used for (supervised) classification tasks, whereas the G-measure (or Fowlkes–Mallows index) is generally used for (unsupervised) clustering tasks? The F-measure is the ...
0
votes
0answers
19 views

AVOD Object Detection Library Using Point Cloud and Image Data

I downloaded and used the AVOD object detection library using the code and instructions available on the following github link. https://github.com/kujason/avod I ...
1
vote
1answer
43 views

Chi-square as evaluation metrics for nonlinear machine learning regression models

I am using machine learning models to predict an ordinal variable (values: 1,2,3,4, and 5) using 7 different features. I posed this as a regression problem, so the final outputs of a model are ...
0
votes
1answer
56 views

Evaluate prediction from multiple classification model

Given that I have data containing images of oranges, apples and pineapples and I want to classify depending on a set of features. Expected that I have completed the model and ready for prediction. ...
1
vote
0answers
67 views

Evaluation of regression models with different evaluations (MSE, variance, VAF etc.)

When comparing several regression models in terms of quality, it seems like most have agreed on the MSE. There are also papers comparing "variance" and "variance accounted for (VAF)". However, there ...
3
votes
1answer
49 views

Evaluating the result of topic modeling in a way that time matters

I have run different topic modeling approach on my data(its clinical data related to Cognitive impairment diseases. we are going to process what thing is important that make it develop to more harsh ...
0
votes
0answers
10 views

Partial dependence plot vs prediction of an averaged row

I have been trying to predict property prices and I want to produce a lat, long partial dependence plot to visualize how much the location of a property affect the price prediction. The formal way of ...
1
vote
1answer
28 views

How to interpret PR and ROC Curve for an unbalanced test set

I have trained a neural network on a dataset, the test set is very unbalanced, ratio between positive examples and negatives is 1:25000. All positive examples are correctly predicted, instead ...
1
vote
1answer
21 views

Clustering evaluation metrics with subquadratic time complexity

It exists many evaluation metrics but often they are quadratic or more on number of data points preventing any application on massive data sets as RAND or Silhouette indexes. For the moment i used : ...
0
votes
0answers
21 views

How to measure model improvement for Recommender Systems in real applications?

In the academia, model 'goodness' for recommendation systems are typically in terms of a loss or metric (i.e. MSE loss, Mean Average Precision). In real world applications, companies would deploy A/B ...
2
votes
3answers
1k views

What is the difference between bootstrapping and cross-validation?

I used to apply K-fold cross-validation for robust evaluation of my machine learning models. But I'm aware of the existence of the bootstrapping method for this purpose as well. However, I cannot see ...
0
votes
1answer
222 views

Calculate average Intersection over Union

I want to have a global IoU metric for each class in a segmentation model with a neural net. The idea is, once the net is trained, doing the forward pass over all training examples an calculate the ...
3
votes
3answers
129 views

Evaluation methods for multi-class classification

I am looking for single-number evaluation method that can be used in multi-class classification tasks that take into account imbalanced data-sets. For instance, ...
6
votes
4answers
3k views

When is precision more important over recall?

Can anyone give me some examples where precision is important and some examples where recall is important ?
1
vote
0answers
100 views

Hyperparameter tuning in multiclass classification problem: which scoring metric?

I'm working with an imbalanced multi class dataset. I try to tune the parameters of a DecisionTreeClassifier, ...
2
votes
2answers
49 views

Size of folds in k-fold cross-validation

When evaluating results using cross-validation, several strategies can be adopted, as using 5 or 10 folds, or doing leave one out cross-validation, as well as doing a 80/20 split. Under which are ...
4
votes
1answer
93 views

Class leaking on validation set

I am quite new in the ML field. I think I correctly understood the information leaking problem during the testing/validation phases but I am struggling to understand some François Chollet statements ...
1
vote
1answer
125 views

Is recall more important than precision for mass mailings?

Say for example, I built a classification model for a mailing campaign that will be applied to 1M records. The positive class for the model would be customers and the negative records would be non-...
0
votes
1answer
260 views

Python - Calculate Cost profitability and benefit of the model

I've this code in Python in order to calculate the precision of my model and to print confusion matrix using Decision Trees Classifier: ...
1
vote
1answer
134 views

What is the advantage of using Dunn index over other metrics for evaluating clustering algorithm? [closed]

There are many metrics to evaluate clustering algorithm like Calinski-Harabaz Index, Dunn index, Rand index, etc. Are there any advantage of using Dunn index over other metrics for evaluating ...
1
vote
0answers
47 views

Performance Metric for topic extraction when there is no ground truth

I am extracting topics from text using a predefined ontology containing 2690 concepts, wordnet(to expand concept terms with their synsets, and other morphological forms of the same word) and lucene to ...
2
votes
1answer
20 views

Find threshold in rate to determine reason for lost customer

I'm not sure if anybody will be able to help with this, or even if I'll be able to explain it well but I am stuck so here goes.... I have a set of customers, some are lost, some are still active. We ...
0
votes
0answers
593 views

LightGBM - Cross validation is performing slightly better with lower iterations than the “best-iteration” used in the model

I have a binary classification LightGBM model that I am running cross validation on (Metric - AUC). I noticed that when I iterate through the AUC for each boosted iteration in the cross validation ...
0
votes
0answers
19 views

Estimating bias when humans are bad

Andrew Ng recommends evaluating the bias of your machine learning algorithm by figuring out how the algorithm's error rate compares to human error rate, with the idea that humans are probably fairly ...
0
votes
0answers
26 views

Is it possible to adjust or fix regression predictions when both error and correlation coefficient are high?

I am using a deep neural network for regression, and when scoring the test predictions of the neural network against the ground truth for the test data, I get these results: ...
3
votes
2answers
163 views

Evaluate new features

How should I evaluate whether new features are effective or not? Should I build a new model with the new features then compare with the old one with the same hyper parameter?
0
votes
0answers
23 views

Benchmark when evaluating performance of a similar documents retrieval?

In what metrics does Stack Overflow, Quora and other QnA sites use to tune their "Questions that may already have your answer" model? Do they use precision/recall described in this. What is a ...
0
votes
1answer
440 views

How to measure F1 score and NMI for clustering task?

I see the authors of this paper are measuring F1 and NMI scores to measure the clustering quality. However, I don't understand the algorithm of how they actually measure it. See the Evaluation Section....
1
vote
0answers
15 views

Do I need to use Bayes to combine a sample's class probability with the performance of the overall model?

For a classification algorithm that gives the predicted probabilities for each class (ie random forest in sklearn), by default the classes are separated with a score of 0.5. After evaluating the ...
0
votes
1answer
28 views
3
votes
1answer
845 views

Splitting hold-out sample and training sample only once?

I have a question related to evaluating out-of-sample predictions. For my research I want to tune two parameters related to Support Vector Machines, and use these optimized parameters to predict the ...
0
votes
1answer
114 views

Clustering documents - how to evaluate results?

I'm using DBSCAN clustering on a set of documents. The documents' content was converted to TF-IDF matrix, and I'd like to find consistent ways to evaluate the clusters when no added information is ...
1
vote
1answer
90 views

How to evaluate sequence to sequence models?

I wonder how to evaluate variable long sequence-to-sequence predictions? Let us say I have the following $Y$ and $\hat{Y}$ $Y = [["1", "2", "2"], ["3", "2", "2"], ["1", "3", "2", "2"]]$ $\hat{Y} = [[...
4
votes
1answer
140 views

How can RL agents be monitored?

My question is about how to monitor RL agents in production. To make the question easier to discuss, here is a use case. Please don't focus on difficulties in implementing such an agent, but rather on ...
1
vote
1answer
84 views

How to evaluate multi label image retrieval model

I'm using a deep hashing model to search most similar images in a database (most similar to the image given as a query). I'm doing this on the coco dataset which has multiple labels per image. I'd ...
1
vote
1answer
427 views

Irregular Precision-Recall Curve

I'd expect that for a precision-recall curve, precision decreases while recall increases monotonically. I have a plot that is not smooth and looks funny. I used scikit learn the values for plotting ...