Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

Filter by
Sorted by
Tagged with
0
votes
0answers
6 views

Probability notation q(y) and q(Y) and its implication to vector functions

The function in question is (from Appendix B, Proof of proposition 2.1 from Posterior Regularization for Structured Latent Variable Models): $$q(\textbf{Z}) = \frac{p_{\theta}(\textbf{Z}|\textbf{X})...
2
votes
1answer
41 views

Why people use precision and not true negative rate (specificity)?

From my experience the standard way to evaluate a classifier is not to check only its accuracy but also its recall and precision. People tend to assume that recall measures the model on positive ...
1
vote
1answer
21 views

Evaluate clustering labels using classification [closed]

I've clustered 500 documents into 7 groups using K-means. Is this reasonable to use classification models to evaluate the clustering model? What I would do is to get these 500 labelled documents using ...
1
vote
0answers
13 views

clustering algorithms' evaluation [closed]

How can I show clustering performance of various clustering algorithms on various datasets using adjusted mutual information and adjusted rand index. for instance, the plot below .
0
votes
0answers
5 views

how to analyze mobile usage and sleep time data

this is my first question on data science forum. I have a little doubt about if I'm asking this question on the right forum therefor any help, edit and change is appreciated. It's been a long time I'm ...
1
vote
1answer
39 views

Select one best model according to accuracy, precision, recall, f1 score and roc score

I have two classifiers that classify the same dataset with these results: ...
1
vote
1answer
15 views

Using precision as a metric - how to gauge if more TP's

So precision is calculated as tp/(tp+fp) But this doesn't seem to be a good way to assess a model as both of the below would give a result of 1? Binary Classification ...
1
vote
1answer
19 views

AUC-PR but there is no recall or precision

Is it possible to have a Precision-Recall curve like this if your recall is zero and your precision is not defined? How do I interpret this? I have checked that all the scores are right and I still ...
2
votes
1answer
19 views

Generate fake model predictions according to desired precision/recall values

Lets assume I generate a random set of target labels for a binary classification with N elements and a certain frequency of the positive class (1), e.g. 10%: ...
0
votes
0answers
13 views

How to calculate mAP for multi-label classification using output predictions?

I have a model which predicts the actions happening in a video clip. Once I get these predictions, I use some rules(set of if-else conditions) to come up with composite labels for eg. ...
0
votes
1answer
32 views

Why performance varies among validate set, public testset and private testset?

When practicing with classical kaggle competitions, such and Titanic, House pricing, and so on, I followed the traditional process that I learned from textbook: split training data into trainig set ...
0
votes
0answers
13 views

After finalizing the hyper-parameters and model using a val set, is it recommended to train the final model using both train and validation sets?

Assume I have all the 3 pre-standardized splits of a dataset (train, val, set), with groundtruth for all 3. Usually many datasets are of this form. Once I have finalized the best configuration of ...
1
vote
1answer
39 views

Using model's prediction score as movement quality evaluator

Let's take the task of evaluating very short dance movements (phrases) using sensor data (accelerometer and gyro from an iPhone device) as an example. If the model's confidence is 100% on a particular ...
0
votes
0answers
72 views

How to train and evaluate model based on inner ordering of data subsets?

I've got a problem which I don't know how to frame properly (what techniques to use, what data structure, etc). Here's a rough definition of it: ...
2
votes
0answers
148 views

How to identify precision, recall, IoU, and mAP in these results for my trained Tensorflow model?

I have trained a Single Shot Detector model (using Tensorflow), and have run the evaluation metrics. However, I am not entirely sure what to make of them. Doing a computer vision literature search, ...
0
votes
1answer
32 views

Interpreting evaluation metrics with threshold/cutoff

I was doing churn prediction for a company. I've got the following results by applying 3 classifier. Model Accuracy AUC Logistic Regression 0.671 0.736 Decision Tree (pruned) 0.681 0.665 Decision ...
0
votes
1answer
48 views

Is sensitivity the same as recall in multiclass classification?

In Wikipedia, it is stated "In binary classification, recall is called sensitivity" under the Recall section. Are they both different in case of multi-class classification?
1
vote
2answers
22 views

Multiple models have extreme differences during evaluation

My dataset has about 100k entries, 6 features, and the label is simple binary classification (about 65% zeros, 35% ones). When I train my dataset on different models: random forest, decision tree, ...
0
votes
1answer
19 views

Evaluate Text-to-speech without Human Involved?

I've explored text-to-speech evaluation matrices and they seem to used Mean Opinion Score (MOS) to evaluate a particular model. This matrice required humans to help to judge the model based on a scale ...
0
votes
0answers
12 views

Why should I use AIC (Akaike information criterion) instead of a metric like RMSE to find the best model?

I have used this AIC metric as a way to find the best SARIMAX model using a grid search to find the values for p,d,q and P,D,Q. I did that because I saw a example of it, but in the end my RMSE result ...
2
votes
3answers
40 views

How are scores calculated for each class of binary classification

The formula for Precision is TP / TP + FP, but how to apply it individually for each class of a binary classification problem, For example here the precision, recall and f1 scores are calculated for ...
1
vote
0answers
16 views

How to compare different forecasting models over different time horizon?

Developed multiple Models with AR, ARIMA, VAR; LSTM , SARIMA. Now, the purpose is to find out which model performs best on a given use case with different time horizons. The time series data is ...
0
votes
0answers
15 views

Simple score function with 4 different indicators

I want to create a function, which returns a value between (0,1) or (-1,1). The result of this function is then used for a boolean decision. E.g. if the value is closer to 0 decision ...
2
votes
1answer
45 views

How to measure the accuracy of an NLP paraphrasing model?

I using the HuggingFace library to do sentence paraphrasing (given an input sentence, the model outputs a paraphrase). How am I supposed to compare the results of two separate models (one trained with ...
0
votes
0answers
27 views

repeated train/test splitting and assessing performance variability

I have a question related to performance variability and how to assess different methods. I want to compare the result of 5 different classifiers on the same dataset (let's say 20 newsgroup dataset). ...
0
votes
0answers
19 views

Calculating effect of entity recognition on a relation extraction system

How can we calculate/formulate the effectiveness of named entity linking (based on P/R/F1 or any other evaluation metrics) on a relation extraction system which accepts the output of ER as its input? ...
0
votes
0answers
25 views

Combine several performance metrics from several datasets

We are developing and evaluating a multi knee/elbow point detection algorithm. For our evaluation, we have 200 sequences of real data. These sequences were annotated manually. For each algorithm and ...
1
vote
1answer
41 views

In general, what are precision, recall, F1 that are reported in papers?

I used classification_report in sklearn library And, the picture below shows evaluation on my model (anomaly detector) In general, what are precision, recall, F1 ...
0
votes
1answer
13 views

How to interprete the feature significance and the evaluation metrics in classification predictive model?

Consider a experiment to predict the Google-Play apps rating using a Random-Forest classifier with scikit-learn in Python. Three attributes 'Free', 'Size' and 'Category' are utilized to predict the ...
1
vote
1answer
42 views

Comparing Multiclass classifiers with "No Answer"-Class

I have three classifiers to classify some words into four classes. Every word that does not fit into any of these four classes gets classified as "No Answer". I would like to compare the ...
1
vote
3answers
25 views

What to do about the failed portion of trained dataset?

I've seen lots of tutorials and papers about this or that model getting some great accuracy score. In this case, let's say 85%. But what I never see is what you are supposed to do with the remaining ...
0
votes
1answer
47 views

Evaluation metric for imbalanced data

Hi I'm a CS graduate student I have a question for AI or data experts. I'm writing a paper My dataset is time-series sensor data and anomaly (positive class) ratio is between 5% and 6% you can see the ...
1
vote
0answers
15 views

Criteria for assessing difficulty of a question

I have a list of questions and how many times they have been answered correctly and incorrectly. Based on this, I applied the formula: ...
1
vote
1answer
43 views

How to test unsupervised learning methods for anomaly detection?

How to test unsupervised learning methods for anomaly detection? I am looking for a test strategy to evaluate my result of my anomaly detection technique? what is your offer more than evaluate with ...
1
vote
1answer
33 views

Evaluation metric for time-series anomaly detection

I have a question for AI or data experts. I'm writing a paper My dataset are time-series sensor data and anomaly ratio is between 5% and 6% 1. For time-series anomaly detection evaluation, which one ...
2
votes
1answer
426 views

How to choose the right threshold for binary classification?

I am currently working on the titanic dataset from Kaggle. The data set is imbalanced with almost 61.5 % negative and 38.5 positive class. I divided my training dataset into 85% train and 15% ...
0
votes
0answers
5 views

Can we use the origional text documnet (which we sumerized) as a reference in ROUGE?

Traditionally, for evaluation, the reference in ROUGE is human generated text (summary) which we compare with system generated text (summary). So consider this, if we generate summaries with different ...
0
votes
0answers
7 views

Search / Multiple Choice System evaluation

I have a DB with N items. My system can output an ID for the item or say N/A (not found). What are different ways to evaluate the performance of such system, and what are the characteristics/tradeoffs ...
1
vote
1answer
98 views

How to construct a test set for an active learning project?

With active learning I hope to keep the annotation effort to a minimum, yet building still a good classifier. My initial starting point is that I have about 20k images which can belong to ten ...
0
votes
0answers
24 views

Evaluate scoring algorithm

I have several large datasets (100k samples) with label 1 / 0 and scoring algorithms that should score each sample, after the scoring of the dataset I sort it by the score. I would like to evaluate ...
1
vote
1answer
33 views

ZeroR as performance baseline for binary classfication model?

It is known that ZeroR model is used predict the majority class in a given data set. Having said that, is ZeroR a suitable performance baseline provided one has a balanced data set (50/50)? If not, ...
2
votes
1answer
56 views

Is it better to use F1 score or AUC metric for imbalanced data classification?

I have a text classification problem, where the "positive" examples are the minority. What metric is better to use for binary classification for this case - F1-score or AUC?
1
vote
0answers
35 views

Permutation importance of features [closed]

This agnostic-model is not well addressed in research papers. I read articles where it was used to test the accuracy of the models, trying to understand the importance of individual features and their ...
0
votes
0answers
27 views

Scoring function for transformers (BERT etc)

While using BERT / transformers for NLP tasks, a major problem faced by us was to detect if the answer returned by model is correct or not, or what is the confidence level of the answer. The ...
0
votes
0answers
39 views

Evaluating a model with different target class distributions between training and testing data

I'm having a bit of an argument about when class imbalances matter when training a classifier, so I was hoping to get some help on understanding a specific concept. Say I have a problem where I want ...
0
votes
0answers
11 views

How to make fair comparison of multi-task RL models if I have unlimited test data?

The data comes from a simulator hence I have the possibility to generate unlimited data. The reward is 0 (no success) and success(1) if episode is successful. Now, the question is what metric to use ...
1
vote
1answer
34 views

How to interpret the rec curve for a regression task?

I am using forest fire dataset and applied neural network model. I tried to generate REC curve, this is how it looks like. Pretty weird!!! I have also applied XGBoost but the REC curve is almost ...
0
votes
0answers
26 views

How to optimize AUPRC for imbalanced data given a precision or recall bias?

My general understanding is that when optimizing a model in an imbalanced class case with a small preferred target class one should optimize first for a model with the best AUPRC (assuming one doesn't ...
3
votes
1answer
202 views

How to create a confusion matrix for one node of a decision tree?

I am doing past papers for my data science exam and was curious about one of the questions. They ask us to create a confusion matrix by hand for one node of a decision tree. I understand how to create ...
2
votes
0answers
29 views

Best way to evaluate interlaced recommendation system results while reducing bias

I already asked this question but I worded it in such a way that it was a completely different question to the one I want to ask. I have not deleted the old question in case someone finds it useful. ...

1
2 3 4 5 6