Questions tagged [model-evaluations]
This tag is meant to be used for questions related to how to evaluate a model performance, not only based on standard metrics, but also in the context of real use case applications. What is a good model might depend on many factors to take into account, to eventually get really useful data science applications.
370
questions
0
votes
0
answers
10
views
What are the most important evaluation metrics for anomaly segmentation?
When people talk about anomaly segmentation models, they often mention evaluation metrics like F1 score, AP, AUROC, and AUPRO. But which one really matters most when comparing models, and why? I'm ...
- 1
1
vote
1
answer
15
views
Reduce mode searching behaviour of VAE
I'm applying VAEs to sections genomic data (haplotypic vcf format, so binary variables), with one model being trained on each section. They each have different layer sizes and weights to better fit ...
- 121
0
votes
0
answers
9
views
İnternal and hold out test for shap
There is an internal test set and a hold out test set. I explain the model with Shap library. Should I use the internal test set or the external test set? What should I do if there is a difference?
- 1
0
votes
0
answers
10
views
How to assess the stability of a DL model, after using k-fold cross-validation for hyperparameter tuning
I've recently completed the training of a deep learning model for a classification task, using a process that involves k-fold cross-validation for hyperparameter tuning
Initially, I have divided my ...
- 1
0
votes
0
answers
12
views
what is the main difference between ROUGE and BLUE?
Both (ROUGE, BLUE) are useful to find the similarity between machine generated summary and reference summary.
what is the main difference?
- 43
0
votes
0
answers
37
views
Why is there a difference in Training Accuracy Output, when the training dataset is the same but the validation dataset is different?
I am looking at the output of a multi-class image segmentation deep learning model. I used U-Net to implement this.
I am confused about why the training accuracies are different for a different ...
0
votes
0
answers
23
views
Difference in the value of evaluation-metric in xgb.train() and predict in R
I have trained a xgboost classifier with a custom metric (f1_xgb), that is, the F1 score. Here the important aspect is that I evaluated the classifier on the test set by setting:
...
2
votes
1
answer
72
views
How to interpret RMSE to evaluate a regression model
I am trying to evaluate a regression model (random forests); my understanding is that R^2 (coefficient of determination) is not a good measure of fitness since my dataset is non-linear. It looks like ...
- 33
0
votes
0
answers
16
views
How to treat "ignore" boxes during object detector evaluation?
I want to evaluate the performance of multiple object detectors for comparison, each of them has been trained on this data. This dataset has images labeled, where each image can have multiple labels. ...
- 121
1
vote
1
answer
21
views
Topic modeling evaluation
I'm working on topic modeling and I have generated clusters with two different methods.
How can I evaluate which method performs better than the other?
- 125
0
votes
0
answers
28
views
PR AUC curve with drop in precision
I have this PR AUC plot, with both PCA and autoencoder related curves having a huge drop of precision in the beginning and then increasing again, with PCA hitting 0 as you can see in the zoomed in ...
0
votes
1
answer
155
views
Understand and compute confidence interval and coefficient of variation for regression model
I would like to better understand the concepts of:
coefficient of variation and confidence interval.
Trivially taking the definitions from wikipedia:
confidence interval (CI)
In frequentist ...
- 1
0
votes
1
answer
28
views
One class classification using Exclusively Positive Examples
I have a dataset consisting only of the positive class, and I want to train a model to identify this data. Is it possible to use one-class classification models for this task? Additionally, how can I ...
0
votes
0
answers
56
views
Actual values vs Preditec values plot
I was working on a project and I got a 0.98 R^2 score on both the training and test data sets and 0.91 training mse and 1.02 test mse, But my Actual values vs Predicted values looks like this, I was ...
0
votes
0
answers
15
views
Comparing a classfiers performance across distinct testing datasets
I have a dataset split into training, validation, and testing sets. I trained this model on the training data and evaluated on the validation and testing sets. Now I have an additional set of data ...
- 73
0
votes
0
answers
22
views
How to choose the correct NN model if the metrics are different in training and test time?
I am trying to build an LSTM model which has a lot of Dropout and Batch Norm Layers. When I run model.fit, the accuracy comes out to around 0.7 on the training data....
0
votes
0
answers
40
views
Does MAPE really penalizes overpredictions?
I read it on many sites, that one of the main "disadvantage" of MAPE is that it penalizes overpredictions, hence it prefers models that are under-predicting.
The main argument is that if we ...
- 25
0
votes
0
answers
12
views
Interpreting large discrepancies between Specificities & the # of Extraneous Variable Models selected by a variable selection algorithm
I am going to preface my question by saying that this problem of interpretation I have run into is in the context of me doing my part as a collaborator on a statistical learning paper for the first ...
- 167
0
votes
0
answers
17
views
Should I use an intercept even if my regression model's r-squared value reduces by a lot?
I'm using Python to create a good linear regression model and am having trouble getting good results for my r-squared value.
A quick rundown of what the data is:
– Sales: This dependent variable ...
0
votes
0
answers
22
views
Model Performance not improving
I am currently working with a GNN (a Graph attention Model) based model and the main task is to do Graph prediction. My model doesnot improve its performance when I change the number of heads or the ...
- 1
1
vote
2
answers
131
views
Xgboost model predicting extreme values for events and non-events | Overfitting
Extreme values are predicted by my trained xgboost classification model in BQML for both events (Y=1) and non-events (Y=0). For all event observations, the model calculates probability scores that ...
- 133
0
votes
0
answers
40
views
XGBoost Classifier Evaluation Confusion on New Dataset Despite High Cross-Validation Scores
I have built an XGBoost classifier model with 90 features, trained on a dataset containing 760k samples. I took great care to separate the labels from the features in both the training and testing ...
- 1
0
votes
1
answer
90
views
Different accuracy scores with sklearn roc_auc_score on same model using sklearn.metrics
Why do these below lines give different outputs while the input is the same?
I need to report these results in paper, but I am unsure which is better and why.
...
- 101
0
votes
1
answer
40
views
how to evaluate a model on our data when the model is imported from a library and thus not trained by us?
The company I work for has deployed a trained rule-based sentiment analyzer model vader to make predictions on customer's attitude. We import the model from nltk library directly, so we didn't train ...
- 3
2
votes
1
answer
276
views
Implementation of spBLEU
I was looking for a way to explore evaluation metrics for language translation models and I came across spBLEU. I can’t find any implementations/examples that would help me start. Does anyone have a ...
- 33
0
votes
0
answers
10
views
Is there a way to artificially manipulate a dataset in order to replace it for one that gives good predictions?
I'm trying to artificially create a dataset for pure educative reasons but I want it to be based in one particular dataset, the problem is that this original dataset don't make good predictions even ...
0
votes
3
answers
265
views
For cross validation should I use training set, or whole dataset?
I'm new to data science and I have a problem understanding what dataset to use when using cross validation for model evaluation.
Let's say I have two models: LogisticRegression and ...
0
votes
1
answer
69
views
eval_metric of XGBoost // ML model in general
Say I am using Xgboost on a binary classification task. eval_metric is one of the model parameter. How should I think about the impact of using different eval_metric(e.g rmse/mae/logloss) in general? ...
- 11
0
votes
1
answer
144
views
How to compare test vs train model performance
When comparing the test vs train model performance to ensure no overfitting (e.g., using AUC ROC as an example), is it better to select the model with the largest test score, or the model with the ...
- 735
-1
votes
1
answer
121
views
Falcon 7B LLM Evaluation using TruLens
The problem I am facing is after defining the prompt template, creating a chain using Langchain, defining the huggingface evaluation module from trulens_eval to check the toxicity of the response and ...
0
votes
0
answers
15
views
How do you figure out how much of a user rating to attribute to each step?
I've got a multi-step pipeline that produces output which is then (sometimes) rated by users. Something like this:
Run sentiment analysis on input
Run intent analysis on input
Choose gating weights ...
-1
votes
1
answer
47
views
Which is the best binary classification model? Train and Test Accuracy are similar
I am building a binary classification model where classes are imbalanced but used SMOTE, I used 4 different models to compare performance and decide which to choose.
They have same train and test ...
- 1
0
votes
0
answers
13
views
One class confusion matrix notation for model evaluation
A one class classification set-up for a set of rules acting as a model, where
each input is a whole dataset
model makes some decision within the dataset for each entry
output is decisions made for ...
- 101
0
votes
1
answer
32
views
model.evaluate gives low results?
i have an image dataset and there are 6300 images with 5 classes . The features extracted and dataset reduced to 256 features. This dataset gives good results(%99) when tested ANN with Backpropagation(...
0
votes
1
answer
173
views
Can I use GridSearchCV.best_score_ for evaluation of model performance?
Scikit-learn page on Grid Search says:
Model selection by evaluating various parameter settings can be seen
as a way to use the labeled data to “train” the parameters of the
grid.
When evaluating the ...
- 103
0
votes
0
answers
13
views
Validation error less than training error
entire project link: Github Repository
In a classification task using Neural Network, I computed the fraction of misclassification as an error. And I am getting a validation error less than the ...
2
votes
2
answers
396
views
Why do residuals of linear regression model need to be normally distributed?
When evaluating the output from a linear/ridge regression model, I have taken the residuals between the predicted and test data. This gives me a normal distribution when I plot this data as a ...
- 23
0
votes
1
answer
89
views
Evaluating overfitting in a logistic regression model
I have developed a logistic regression model for a classification problem and obtained an AUC (Area Under the Curve) score of approximately 0.9. The model was estimated by splitting the available data ...
- 1
0
votes
1
answer
25
views
How to evaluate machine translations of long documents?
I'm using Python and I want to compare the output of two machine-translation (ish) systems. Most of the tools seem to be focused on sentence-by-sentence evaluation. Either I get memory blow-ups with ...
- 101
0
votes
0
answers
33
views
Which loss function should I use if multiple values are correct?
My task is to create a QA-model. I give it a context and a question that it should answer. The answer is usually one word, so a very simplified input would be e.g.
Context: "Max eats a banana. ...
0
votes
0
answers
25
views
Comparing Models based on two different sample sets of a single data set
I want to compare the performance of two different ML models like M1 and M2. I have a very huge data set and having two different downsampling of this data set, call them S1 and S2. Can I compare the ...
- 113
0
votes
0
answers
20
views
torchmetrics BinaryMatthewsCorrCoef outputs 0 if target and prediction contains only one case either positive or negative case
I stared using MCC(Matthew's correlation coefficient) metric. But getting unexpected values when the given target and pred contains only one case either positive or negative (case - 1), The output of ...
- 1
0
votes
1
answer
1k
views
How to correctly measure the inference time and FLOPs of a model?
For some reason, I can’t find built-in solutions (not really?) in keras and tensorflow, while on the site
https://keras.io/api/applications/ they provide Time (ms) per inference step (CPU), but for ...
1
vote
1
answer
32
views
Is there such thing as dataset imrovement?
I know that we can use explained machine learning to find why a model chose a certain classification.
I wonder if there is a way I can find which features are going to improve my current model.
I will ...
- 549
0
votes
0
answers
295
views
Multilabel metrics: micro vs. macro vs. weighted vs. samples?
I'm working on a multilabel classification problem; there are $N$ classes and each example can belong to $[0, N]$ of those classes. Below you can see the precision and recall computed using various ...
0
votes
0
answers
16
views
why performance values for passive aggressive model changing every time?
i havq question, that am a little confused about PassiveAggressive performance values after i fitted tf-idf vectorization on it(bigram and trigram), the values are changed every time i run the code ...
- 1
1
vote
1
answer
108
views
Benchmarking LLMs on technical questions
There are several existing benchmark sets to evaluate the performance of large language models on natural-language comprehension tasks, such as CoQA, LAMBDA, HELLASWAG, LogiQA.
I'm interested in ...
- 139
0
votes
1
answer
75
views
How do I know If my regression model is underfitting?
How do we evaluate the performance of a regression model with a certain RMSE given that a domain knowledge performance metric is not present?
Maybe MAPE is one way of comparing the performance of my ...
- 31
0
votes
0
answers
19
views
What are some approaches to improve the classification of ONE particular instance of interest?
I'd like to know if there are some methods to correct a specific misclassified instance of interest (e.g. in a Logistic Regression or Random Forest).
Like maybe increasing the error for that ...
- 11
1
vote
1
answer
62
views
Imbalanced performance metrics in binary classification
I am developing a binary classification model using sklearn pipeline for preprocessing and a soft voting classifier (Adaboost and Extratrees with 50 estimators). The dataset (3 million rows) contains ...
- 23