Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [evaluation]

The tag has no usage guidance.

1
vote
2answers
41 views

Statistical test for machine learning

I want to prove that my proposed machine learning algorithm (prop_ml) is better than other baseline algorithms (ml_1, ml_2, ml_3) when given a small number of data for training. What I've done is to ...
1
vote
0answers
11 views

very large difference between cross_val and (multiple) r2 model evaluation

I did submit my first kaggle kernel, on the avocado dataset kernel link, I treated it like I should predict the avocado price so I splitted the dataset in a train & test set, fitted the model and ...
0
votes
0answers
13 views

How to set score for a set of evaluations based on the number of responses?

Our company has an survey module where our clients evaluate our employees. Each employee can have from 1 to N evaluations being N equal to the total number of clients. The evaluation has several ...
2
votes
0answers
20 views

Scikit-learn average_precision_score() vs. auc score of precision_recall_curve()

I've been searching around for an explanation to this, and haven't come across one yet- in scikit-learn, when I compute the auc() of the ...
2
votes
0answers
6 views

What would be the main and essential criteria for evaluating auto-sklearn library ?

I m running experiments using benchmark datasets with auto-sklearn to see how its performance is different to the standard sklearn library, Since automl does an exhaustive search over parameters and ...
2
votes
1answer
42 views

Macro- or micro-average for imbalanced class problems

The question of whether to use macro- or micro-averages when the data is imbalanced comes up all the time. Some googling shows that many bloggers tend to say that micro-average is the preferred way ...
4
votes
1answer
57 views

Why is the F-measure preferred for classification tasks?

Why is the F-measure usually used for (supervised) classification tasks, whereas the G-measure (or Fowlkes–Mallows index) is generally used for (unsupervised) clustering tasks? The F-measure is the ...
0
votes
0answers
11 views

AVOD Object Detection Library Using Point Cloud and Image Data

I downloaded and used the AVOD object detection library using the code and instructions available on the following github link. https://github.com/kujason/avod I ...
1
vote
1answer
30 views

Chi-square as evaluation metrics for nonlinear machine learning regression models

I am using machine learning models to predict an ordinal variable (values: 1,2,3,4, and 5) using 7 different features. I posed this as a regression problem, so the final outputs of a model are ...
0
votes
1answer
51 views

Evaluate prediction from multiple classification model

Given that I have data containing images of oranges, apples and pineapples and I want to classify depending on a set of features. Expected that I have completed the model and ready for prediction. ...
1
vote
0answers
62 views

Evaluation of regression models with different evaluations (MSE, variance, VAF etc.)

When comparing several regression models in terms of quality, it seems like most have agreed on the MSE. There are also papers comparing "variance" and "variance accounted for (VAF)". However, there ...
3
votes
1answer
49 views

Evaluating the result of topic modeling in a way that time matters

I have run different topic modeling approach on my data(its clinical data related to Cognitive impairment diseases. we are going to process what thing is important that make it develop to more harsh ...
0
votes
0answers
8 views

Partial dependence plot vs prediction of an averaged row

I have been trying to predict property prices and I want to produce a lat, long partial dependence plot to visualize how much the location of a property affect the price prediction. The formal way of ...
1
vote
1answer
23 views

How to interpret PR and ROC Curve for an unbalanced test set

I have trained a neural network on a dataset, the test set is very unbalanced, ratio between positive examples and negatives is 1:25000. All positive examples are correctly predicted, instead ...
1
vote
1answer
20 views

Clustering evaluation metrics with subquadratic time complexity

It exists many evaluation metrics but often they are quadratic or more on number of data points preventing any application on massive data sets as RAND or Silhouette indexes. For the moment i used : ...
0
votes
0answers
20 views

How to measure model improvement for Recommender Systems in real applications?

In the academia, model 'goodness' for recommendation systems are typically in terms of a loss or metric (i.e. MSE loss, Mean Average Precision). In real world applications, companies would deploy A/B ...
2
votes
3answers
342 views

What is the difference between bootstrapping and cross-validation?

I used to apply K-fold cross-validation for robust evaluation of my machine learning models. But I'm aware of the existence of the bootstrapping method for this purpose as well. However, I cannot see ...
0
votes
1answer
129 views

Calculate average Intersection over Union

I want to have a global IoU metric for each class in a segmentation model with a neural net. The idea is, once the net is trained, doing the forward pass over all training examples an calculate the ...
3
votes
3answers
95 views

Evaluation methods for multi-class classification

I am looking for single-number evaluation method that can be used in multi-class classification tasks that take into account imbalanced data-sets. For instance, ...
6
votes
4answers
2k views

When is precision more important over recall?

Can anyone give me some examples where precision is important and some examples where recall is important ?
1
vote
0answers
57 views

Hyperparameter tuning in multiclass classification problem: which scoring metric?

I'm working with an imbalanced multi class dataset. I try to tune the parameters of a DecisionTreeClassifier, ...
2
votes
2answers
45 views

Size of folds in k-fold cross-validation

When evaluating results using cross-validation, several strategies can be adopted, as using 5 or 10 folds, or doing leave one out cross-validation, as well as doing a 80/20 split. Under which are ...
4
votes
1answer
75 views

Class leaking on validation set

I am quite new in the ML field. I think I correctly understood the information leaking problem during the testing/validation phases but I am struggling to understand some François Chollet statements ...
1
vote
1answer
104 views

Is recall more important than precision for mass mailings?

Say for example, I built a classification model for a mailing campaign that will be applied to 1M records. The positive class for the model would be customers and the negative records would be non-...
0
votes
1answer
166 views

Python - Calculate Cost profitability and benefit of the model

I've this code in Python in order to calculate the precision of my model and to print confusion matrix using Decision Trees Classifier: ...
1
vote
1answer
64 views

What is the advantage of using Dunn index over other metrics for evaluating clustering algorithm? [closed]

There are many metrics to evaluate clustering algorithm like Calinski-Harabaz Index, Dunn index, Rand index, etc. Are there any advantage of using Dunn index over other metrics for evaluating ...
1
vote
0answers
41 views

Performance Metric for topic extraction when there is no ground truth

I am extracting topics from text using a predefined ontology containing 2690 concepts, wordnet(to expand concept terms with their synsets, and other morphological forms of the same word) and lucene to ...
2
votes
1answer
19 views

Find threshold in rate to determine reason for lost customer

I'm not sure if anybody will be able to help with this, or even if I'll be able to explain it well but I am stuck so here goes.... I have a set of customers, some are lost, some are still active. We ...
0
votes
0answers
526 views

LightGBM - Cross validation is performing slightly better with lower iterations than the “best-iteration” used in the model

I have a binary classification LightGBM model that I am running cross validation on (Metric - AUC). I noticed that when I iterate through the AUC for each boosted iteration in the cross validation ...
0
votes
0answers
19 views

Estimating bias when humans are bad

Andrew Ng recommends evaluating the bias of your machine learning algorithm by figuring out how the algorithm's error rate compares to human error rate, with the idea that humans are probably fairly ...
0
votes
0answers
26 views

Is it possible to adjust or fix regression predictions when both error and correlation coefficient are high?

I am using a deep neural network for regression, and when scoring the test predictions of the neural network against the ground truth for the test data, I get these results: ...
3
votes
2answers
144 views

Evaluate new features

How should I evaluate whether new features are effective or not? Should I build a new model with the new features then compare with the old one with the same hyper parameter?
0
votes
0answers
20 views

Benchmark when evaluating performance of a similar documents retrieval?

In what metrics does Stack Overflow, Quora and other QnA sites use to tune their "Questions that may already have your answer" model? Do they use precision/recall described in this. What is a ...
0
votes
1answer
363 views

How to measure F1 score and NMI for clustering task?

I see the authors of this paper are measuring F1 and NMI scores to measure the clustering quality. However, I don't understand the algorithm of how they actually measure it. See the Evaluation Section....
1
vote
0answers
14 views

Do I need to use Bayes to combine a sample's class probability with the performance of the overall model?

For a classification algorithm that gives the predicted probabilities for each class (ie random forest in sklearn), by default the classes are separated with a score of 0.5. After evaluating the ...
0
votes
1answer
27 views
3
votes
1answer
595 views

Splitting hold-out sample and training sample only once?

I have a question related to evaluating out-of-sample predictions. For my research I want to tune two parameters related to Support Vector Machines, and use these optimized parameters to predict the ...
0
votes
1answer
86 views

Clustering documents - how to evaluate results?

I'm using DBSCAN clustering on a set of documents. The documents' content was converted to TF-IDF matrix, and I'd like to find consistent ways to evaluate the clusters when no added information is ...
1
vote
1answer
84 views

How to evaluate sequence to sequence models?

I wonder how to evaluate variable long sequence-to-sequence predictions? Let us say I have the following $Y$ and $\hat{Y}$ $Y = [["1", "2", "2"], ["3", "2", "2"], ["1", "3", "2", "2"]]$ $\hat{Y} = [[...
4
votes
1answer
140 views

How can RL agents be monitored?

My question is about how to monitor RL agents in production. To make the question easier to discuss, here is a use case. Please don't focus on difficulties in implementing such an agent, but rather on ...
1
vote
1answer
69 views

How to evaluate multi label image retrieval model

I'm using a deep hashing model to search most similar images in a database (most similar to the image given as a query). I'm doing this on the coco dataset which has multiple labels per image. I'd ...
1
vote
1answer
316 views

Irregular Precision-Recall Curve

I'd expect that for a precision-recall curve, precision decreases while recall increases monotonically. I have a plot that is not smooth and looks funny. I used scikit learn the values for plotting ...
-1
votes
1answer
158 views

What are the drawbacks of V-measure clustering evaluation method?

What are the drawbacks of V-measure clustering evaluation method? For evaluating what clustering algorithms, is the V-measure evaluation method suitable?
0
votes
1answer
55 views

How to get an intuitive value for regression module evaluation?

For regression module evaluation, I think only the MAE (Mean absolute error) value is not objective or practical. Consider following situations: A MAE=1 while ...
0
votes
2answers
81 views

recommender system: how to compare different scores when calculated individually?

I am building a small recommender system which aims at recommending ~10 products to customers. Instead of using a multi-label classification model, I have opted to build a separate scoring model for ...
0
votes
1answer
70 views

Metrics show badly performing model for multiclass

So I have a model that I am training on a multiclass (30-40 classes) imbalanced data set (smallest class 4000 samples, largest 14 million). The data consists of strings and I extract unigram and ...
-1
votes
1answer
332 views

In sklearn's classification report, is f1 the best accuracy measure?

In the classification_report provided by sklearn, which score should I look at to make the best determination of the accuracy of my model? ...
6
votes
1answer
2k views

How many features to sample using Random Forests

The Wikipedia page which quotes "The Elements of Statistical Learning" says: Typically, for a classification problem with $p$ features, $\lfloor \sqrt{p}\rfloor$ features are used in each split. ...
1
vote
0answers
11 views

What does NIST information weights refer to?

NIST is a metric used to measure the goodness of translation. In the paper, Doddington (2002) introduce the notion of "Information weights" Information weights were computed using N-gram counts ...
1
vote
2answers
432 views

Why exactly using a test set for model evaluation is a bad idea?

I don't understand why using the test set for model evaluation is a bad idea. I completely understand why you should not use your test set to train your model (because in that case, you would be ...