Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

How does exactly eval_set and RandomizedSearchCV work for LightGBM?

How does RandomizedSearchCV form the validation sets, while I also defined an evaluation set for LGBM? Is it formed from the train set I gave or how does the evaluation set comes into the validation? ...
user avatar
1 vote
1 answer
16 views

Evaluating Models that Return Percentage Present of Multiple Classes

If there is a model that returns a vector of the amount of different classes present in the data as percentages, what would be a good way to evaluate it (with charts and/or statistics)? Say, for ...
user avatar
  • 13
0 votes
0 answers
10 views

Evaluation startegies for online machine learning

I am intending to investigate the performance online Vs offline (batch) learning approaches in a research. the use-case is condition monitoring (classification problem for fault-detection in a non-...
user avatar
0 votes
0 answers
15 views

How to draw a ROC curve for each Fold in cross validation in R

I am evaluating my model using K fold cross validation and I would like to draw a ROC curve for each of the folds and show them ALL TOGETHER. I'm using the R programming language and I'm going to ...
user avatar
  • 302
3 votes
2 answers
531 views

How to evaluate when recommender systems are influencing behavior?

Consider a recommender system which sends discount coupons for cakes to visitors on some website. There are 2 cases: good case: when a customer visits the website with no intent of buying a cake, but ...
user avatar
  • 131
1 vote
1 answer
24 views

When to prioritize accuracy over precision?

I am working on a simple SVM project for the prediction of hepatitis c. I got my dataset from kaggle. When dealing with null values, I tried two ways, firstly by dropping data with null values, second ...
user avatar
0 votes
0 answers
16 views

How to draw the ROC curve of a classifier evaluated with cross validation

I was wondering if anyone knows without with the R programming language, given a classifier evaluated by k fold cross validation, I can draw each of the ROC curves that are generated in each fold of ...
user avatar
  • 302
0 votes
0 answers
9 views

Orange v3.32: Accuracy and precision not showing up

As explained in the orangehelp files the test and score widget would provide an accuracy colum like "CA". I only have MSE RMSE MAE and R2 besides the times. Furthermore, the predictions ...
user avatar
  • 1
1 vote
1 answer
27 views

Need term or method name for evaluation of CNN without ground truth using e.g. a regression model

I have the following problem, I have trained a CNN and I can evaluate the network in-sample. I want to use the trained model for the class prediction of images for which I have no ground truth. ...
user avatar
3 votes
2 answers
102 views

How to assign costs to the confusion matrix

I am trying to assign costs to the confusion matrix. That is, in my problem, a FP does not have the same cost as a FN, so I want to assign to these cases a cost "x" so that the algorithm ...
user avatar
  • 302
0 votes
0 answers
13 views

Inference speed of ReLU networks

I'm fairly new in the topic, and I was wondering whether some of you can point to existing works in which the inference of deep neural networks with ReLU activation functions is tested on GPUs as a ...
user avatar
1 vote
0 answers
23 views

Best metric to evaluate model probabilities

i'm trying to create ML model for binary classification problem with balanced dataset and i care mostly about probabilities. I was trying to search web and i find only advices to use AUC or logloss ...
user avatar
0 votes
1 answer
21 views

What metrics work well in unbalanced assemblies?

I wanted to know if there are some metrics that work well when working with an unbalanced dataset. I know that accuracy is a very bad metric when evaluating a classifier when the data is unbalanced ...
user avatar
  • 302
1 vote
0 answers
62 views

How to draw each ROC curve of an SVM model with cross validation

I would like to make a graph like the following in python: That is, one curve for each fold. I have the following code where I use an SVM model to classify some data ...
user avatar
  • 302
0 votes
1 answer
20 views

Ideal Windows Size in Pk Evaluation Metric

I am very new to nlp. I am doing a text segmentation task and for evaluating my model I need to calculate Pk and Windiff scores. My question is what is the ideal value for window size (k) for Pk score ...
user avatar
  • 1
1 vote
1 answer
17 views

How to determine the "total number of relevant documents" in calculatiion of Recall in Precision and Recall if it's not known? Can it be estimated?

On Wikipedia there is a practical example of calculating Precision and Recall: When a search engine returns 30 pages, only 20 of which are relevant, while failing to return 40 additional relevant ...
user avatar
  • 11
0 votes
0 answers
9 views

How are ROC curves constructed? [duplicate]

I would like to understand how to build a ROC curve of a model. For example, if we would like to draw it by hand, what steps should we do? Thank you.
user avatar
  • 302
0 votes
0 answers
7 views

Difference between Jackknife vs bootsrap vs cross validation

I have doubts about the differences between these three methods and I would like to clarify the following: Main differences Advantages of one over the other Context of use of each method etc... If ...
user avatar
  • 302
12 votes
3 answers
4k views

What are the disadvantages of accuracy?

I have been reading about evaluating a model with accuracy only and I have found some disadvantages. Among them, I read that it equates all errors. How could this problem be solved? Maybe assigning ...
user avatar
  • 302
1 vote
1 answer
23 views

Bias-variance trade-off and model evaluation

Suppose that we have train a model (as defined by its hyperparameters) and we evaluated it on a test set using some performance metric (say $R^2$). If we now train the same model (as defined by its ...
user avatar
  • 131
1 vote
2 answers
43 views

How are the confidence intervals of a model interpreted?

I am doing some work with R and after obtaining the confusion matrix I have obtained the following metrics corresponding to a logistic regression: ...
user avatar
  • 302
0 votes
1 answer
190 views

model.fit vs model.evaluate gives different results?

The following is a small snippet of the code, but I'm trying to understand the results of model.fit with train and test dataset vs the model.evaluate results. I'm not sure if they do not match up or ...
user avatar
  • 101
0 votes
0 answers
9 views

Soft-clustering evaluation with multiple labels

I have an article clustering problem ( the articles are encoded with T5 so I technically have vectors) where each one can have multiple topics as labels(the set of labels is unbounded). I did soft ...
user avatar
0 votes
1 answer
9 views

How to extract MSEP or RMSEP from lassoCV?

I'm doing lasso and ridge regression in R with the package chemometrics. With ridgeCV it is easy to extract the SEP and MSEP values by ...
user avatar
  • 5
2 votes
0 answers
89 views

Is there a Mean Average Recall for Item Retrieval/ Recommendation Systems?

Mean Average Precision for Information retrieval is computed using Average Precision @ k (AP@k). AP@k is measured by first computing Precision @ k (P@k) and then averaging the P@k only for the k's ...
user avatar
  • 21
0 votes
0 answers
31 views

Repeatability tests for machine learning models (in the sense of measurement system analysis)

For analyzing a machine learning model, we usually calculate the model performance metrics (such as accuracy...) and during validation step make sure that the model has not overfitted. We can consider ...
user avatar
  • 1
0 votes
0 answers
27 views

How can I interpret my rho risk values when performing probabilistic time series forecasting?

I am currently exploring different probabilistic time series forecasting models for car sales data and have planned to evaluate the probabilistic forecasts with the metrics rho-risk as described on ...
user avatar
1 vote
1 answer
27 views

Looking for in depth knowledge in evalution metric

I am dealing with an unbalanced dataset. The total instances in my dataset is 1273 and the Yes class is 174 and No class is 1099. So the unbalance ratio is like 1:6. Now I know ...
user avatar
  • 137
2 votes
1 answer
25 views

Evaluate two Recommender models trained with different data

Suppose you are given two Recommender Systems to evaluate, A and B. Model A is trained with ...
user avatar
0 votes
0 answers
14 views

What is the metric to evaluate directional performance in regression models?

Not sure if the title is confusing or not. Here is an example: Let's say we build a regression model to predict the housing price. However, instead of caring about the prediction accuracy where MAE, ...
user avatar
  • 101
1 vote
2 answers
79 views

Evaluation Metric for Imbalanced and Ordinal Classification

I'm looking for an ML evaluation metric that would work well with imbalanced and ordinal multiclass datasets: Imagine you want to predict the severity of a disease that has 4 grades of severity where ...
user avatar
1 vote
0 answers
14 views

Probability notation q(y) and q(Y) and its implication to vector functions

The function in question is (from Appendix B, Proof of proposition 2.1 from Posterior Regularization for Structured Latent Variable Models): $$q(\textbf{Z}) = \frac{p_{\theta}(\textbf{Z}|\textbf{X})...
user avatar
  • 131
3 votes
1 answer
55 views

Why people use precision and not true negative rate (specificity)?

From my experience the standard way to evaluate a classifier is not to check only its accuracy but also its recall and precision. People tend to assume that recall measures the model on positive ...
user avatar
1 vote
1 answer
26 views

Evaluate clustering labels using classification [closed]

I've clustered 500 documents into 7 groups using K-means. Is this reasonable to use classification models to evaluate the clustering model? What I would do is to get these 500 labelled documents using ...
user avatar
1 vote
0 answers
16 views

clustering algorithms' evaluation [closed]

How can I show clustering performance of various clustering algorithms on various datasets using adjusted mutual information and adjusted rand index. for instance, the plot below .
user avatar
  • 11
0 votes
0 answers
5 views

how to analyze mobile usage and sleep time data

this is my first question on data science forum. I have a little doubt about if I'm asking this question on the right forum therefor any help, edit and change is appreciated. It's been a long time I'm ...
user avatar
1 vote
1 answer
109 views

Select one best model according to accuracy, precision, recall, f1 score and roc score

I have two classifiers that classify the same dataset with these results: ...
user avatar
  • 13
1 vote
1 answer
16 views

Using precision as a metric - how to gauge if more TP's

So precision is calculated as tp/(tp+fp) But this doesn't seem to be a good way to assess a model as both of the below would give a result of 1? Binary Classification ...
user avatar
1 vote
1 answer
37 views

AUC-PR but there is no recall or precision

Is it possible to have a Precision-Recall curve like this if your recall is zero and your precision is not defined? How do I interpret this? I have checked that all the scores are right and I still ...
user avatar
2 votes
1 answer
32 views

Generate fake model predictions according to desired precision/recall values

Lets assume I generate a random set of target labels for a binary classification with N elements and a certain frequency of the positive class (1), e.g. 10%: ...
user avatar
0 votes
0 answers
24 views

How to calculate mAP for multi-label classification using output predictions?

I have a model which predicts the actions happening in a video clip. Once I get these predictions, I use some rules(set of if-else conditions) to come up with composite labels for eg. ...
user avatar
0 votes
1 answer
40 views

Why performance varies among validate set, public testset and private testset?

When practicing with classical kaggle competitions, such and Titanic, House pricing, and so on, I followed the traditional process that I learned from textbook: split training data into trainig set ...
user avatar
0 votes
0 answers
17 views

After finalizing the hyper-parameters and model using a val set, is it recommended to train the final model using both train and validation sets?

Assume I have all the 3 pre-standardized splits of a dataset (train, val, set), with groundtruth for all 3. Usually many datasets are of this form. Once I have finalized the best configuration of ...
user avatar
  • 131
1 vote
1 answer
39 views

Using model's prediction score as movement quality evaluator

Let's take the task of evaluating very short dance movements (phrases) using sensor data (accelerometer and gyro from an iPhone device) as an example. If the model's confidence is 100% on a particular ...
user avatar
  • 111
0 votes
0 answers
72 views

How to train and evaluate model based on inner ordering of data subsets?

I've got a problem which I don't know how to frame properly (what techniques to use, what data structure, etc). Here's a rough definition of it: ...
user avatar
  • 125
2 votes
0 answers
239 views

How to identify precision, recall, IoU, and mAP in these results for my trained Tensorflow model?

I have trained a Single Shot Detector model (using Tensorflow), and have run the evaluation metrics. However, I am not entirely sure what to make of them. Doing a computer vision literature search, ...
user avatar
  • 141
0 votes
1 answer
73 views

Interpreting evaluation metrics with threshold/cutoff

I was doing churn prediction for a company. I've got the following results by applying 3 classifier. Model Accuracy AUC Logistic Regression 0.671 0.736 Decision Tree (pruned) 0.681 0.665 Decision ...
user avatar
  • 23
0 votes
1 answer
111 views

Is sensitivity the same as recall in multiclass classification?

In Wikipedia, it is stated "In binary classification, recall is called sensitivity" under the Recall section. Are they both different in case of multi-class classification?
user avatar
1 vote
2 answers
25 views

Multiple models have extreme differences during evaluation

My dataset has about 100k entries, 6 features, and the label is simple binary classification (about 65% zeros, 35% ones). When I train my dataset on different models: random forest, decision tree, ...
user avatar
  • 13
0 votes
1 answer
21 views

Evaluate Text-to-speech without Human Involved?

I've explored text-to-speech evaluation matrices and they seem to used Mean Opinion Score (MOS) to evaluate a particular model. This matrice required humans to help to judge the model based on a scale ...
user avatar

1
2 3 4 5 6