Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [model-evaluations]

This tag is meant to be used for questions related to how to evaluate a model performance, not only based on standard metrics, but also in the context of real use case applications. What is a good model might depend on many factors to take into account, to eventually get really useful data science applications.

Filter by
Sorted by
Tagged with
1 vote
1 answer
36 views

How to measure different models' feature importance using a generic and common standard?

I want to measure the feature importance of a series of models after training them. Most models have some built-in APIs that allow me to access their feature importance, but as far as I know, these ...
Yuuya's user avatar
  • 11
0 votes
0 answers
13 views

How to use cross validation to select/evaluate model with probability score as the output?

Initially I was evaluating my models using cross_val with out-of-pocket metrics such as precision, recall, f1 score, etc, or with my own metrics defined in ...
szheng's user avatar
  • 21
1 vote
1 answer
92 views

How do I work with time-series data of temperature?

So I have some equipment temperature and i have outside temperature (both are collected daily) and I want to predict the equipment temperature. However, I'm new to this and unsure about which model to ...
Ria's user avatar
  • 11
0 votes
0 answers
15 views

What are the most important evaluation metrics for anomaly segmentation?

When people talk about anomaly segmentation models, they often mention evaluation metrics like F1 score, AP, AUROC, and AUPRO. But which one really matters most when comparing models, and why? I'm ...
Mosh Geb's user avatar
1 vote
1 answer
15 views

Reduce mode searching behaviour of VAE

I'm applying VAEs to sections genomic data (haplotypic vcf format, so binary variables), with one model being trained on each section. They each have different layer sizes and weights to better fit ...
Whitehot's user avatar
  • 121
0 votes
0 answers
9 views

İnternal and hold out test for shap

There is an internal test set and a hold out test set. I explain the model with Shap library. Should I use the internal test set or the external test set? What should I do if there is a difference?
Nemo's user avatar
  • 1
0 votes
0 answers
10 views

How to assess the stability of a DL model, after using k-fold cross-validation for hyperparameter tuning

I've recently completed the training of a deep learning model for a classification task, using a process that involves k-fold cross-validation for hyperparameter tuning Initially, I have divided my ...
o'hara's user avatar
  • 1
0 votes
0 answers
13 views

what is the main difference between ROUGE and BLUE?

Both (ROUGE, BLUE) are useful to find the similarity between machine generated summary and reference summary. what is the main difference?
Tovlk's user avatar
  • 43
0 votes
0 answers
37 views

Why is there a difference in Training Accuracy Output, when the training dataset is the same but the validation dataset is different?

I am looking at the output of a multi-class image segmentation deep learning model. I used U-Net to implement this. I am confused about why the training accuracies are different for a different ...
user10529827's user avatar
0 votes
0 answers
26 views

Difference in the value of evaluation-metric in xgb.train() and predict in R

I have trained a xgboost classifier with a custom metric (f1_xgb), that is, the F1 score. Here the important aspect is that I evaluated the classifier on the test set by setting: ...
user159339's user avatar
2 votes
1 answer
81 views

How to interpret RMSE to evaluate a regression model

I am trying to evaluate a regression model (random forests); my understanding is that R^2 (coefficient of determination) is not a good measure of fitness since my dataset is non-linear. It looks like ...
Shawn's user avatar
  • 35
0 votes
0 answers
21 views

How to treat "ignore" boxes during object detector evaluation?

I want to evaluate the performance of multiple object detectors for comparison, each of them has been trained on this data. This dataset has images labeled, where each image can have multiple labels. ...
hafiz031's user avatar
  • 121
1 vote
1 answer
23 views

Topic modeling evaluation

I'm working on topic modeling and I have generated clusters with two different methods. How can I evaluate which method performs better than the other?
user5520049's user avatar
0 votes
0 answers
30 views

PR AUC curve with drop in precision

I have this PR AUC plot, with both PCA and autoencoder related curves having a huge drop of precision in the beginning and then increasing again, with PCA hitting 0 as you can see in the zoomed in ...
GabrielPast's user avatar
0 votes
1 answer
156 views

Understand and compute confidence interval and coefficient of variation for regression model

I would like to better understand the concepts of: coefficient of variation and confidence interval. Trivially taking the definitions from wikipedia: confidence interval (CI) In frequentist ...
Cata's user avatar
  • 1
0 votes
1 answer
29 views

One class classification using Exclusively Positive Examples

I have a dataset consisting only of the positive class, and I want to train a model to identify this data. Is it possible to use one-class classification models for this task? Additionally, how can I ...
user13241829's user avatar
0 votes
0 answers
62 views

Actual values vs Preditec values plot

I was working on a project and I got a 0.98 R^2 score on both the training and test data sets and 0.91 training mse and 1.02 test mse, But my Actual values vs Predicted values looks like this, I was ...
Ario Alavi's user avatar
0 votes
0 answers
15 views

Comparing a classfiers performance across distinct testing datasets

I have a dataset split into training, validation, and testing sets. I trained this model on the training data and evaluated on the validation and testing sets. Now I have an additional set of data ...
tensormoby's user avatar
0 votes
0 answers
22 views

How to choose the correct NN model if the metrics are different in training and test time?

I am trying to build an LSTM model which has a lot of Dropout and Batch Norm Layers. When I run model.fit, the accuracy comes out to around 0.7 on the training data....
Jeffrey Davidson's user avatar
0 votes
0 answers
42 views

Does MAPE really penalizes overpredictions?

I read it on many sites, that one of the main "disadvantage" of MAPE is that it penalizes overpredictions, hence it prefers models that are under-predicting. The main argument is that if we ...
morqueatsz's user avatar
0 votes
0 answers
12 views

Interpreting large discrepancies between Specificities & the # of Extraneous Variable Models selected by a variable selection algorithm

I am going to preface my question by saying that this problem of interpretation I have run into is in the context of me doing my part as a collaborator on a statistical learning paper for the first ...
Marlen's user avatar
  • 167
0 votes
0 answers
18 views

Should I use an intercept even if my regression model's r-squared value reduces by a lot?

I'm using Python to create a good linear regression model and am having trouble getting good results for my r-squared value. A quick rundown of what the data is: – Sales: This dependent variable ...
Python Student's user avatar
0 votes
0 answers
22 views

Model Performance not improving

I am currently working with a GNN (a Graph attention Model) based model and the main task is to do Graph prediction. My model doesnot improve its performance when I change the number of heads or the ...
Susan's user avatar
  • 1
1 vote
2 answers
139 views

Xgboost model predicting extreme values for events and non-events | Overfitting

Extreme values are predicted by my trained xgboost classification model in BQML for both events (Y=1) and non-events (Y=0). For all event observations, the model calculates probability scores that ...
Scott Grammilo's user avatar
0 votes
0 answers
47 views

XGBoost Classifier Evaluation Confusion on New Dataset Despite High Cross-Validation Scores

I have built an XGBoost classifier model with 90 features, trained on a dataset containing 760k samples. I took great care to separate the labels from the features in both the training and testing ...
oklen's user avatar
  • 1
0 votes
1 answer
115 views

Different accuracy scores with sklearn roc_auc_score on same model using sklearn.metrics

Why do these below lines give different outputs while the input is the same? I need to report these results in paper, but I am unsure which is better and why. ...
Adnan Ali's user avatar
  • 101
0 votes
1 answer
41 views

how to evaluate a model on our data when the model is imported from a library and thus not trained by us?

The company I work for has deployed a trained rule-based sentiment analyzer model vader to make predictions on customer's attitude. We import the model from nltk library directly, so we didn't train ...
Shelby's user avatar
  • 3
2 votes
1 answer
320 views

Implementation of spBLEU

I was looking for a way to explore evaluation metrics for language translation models and I came across spBLEU. I can’t find any implementations/examples that would help me start. Does anyone have a ...
Prithvi's user avatar
  • 33
0 votes
0 answers
10 views

Is there a way to artificially manipulate a dataset in order to replace it for one that gives good predictions?

I'm trying to artificially create a dataset for pure educative reasons but I want it to be based in one particular dataset, the problem is that this original dataset don't make good predictions even ...
CinfaCinfa's user avatar
0 votes
3 answers
311 views

For cross validation should I use training set, or whole dataset?

I'm new to data science and I have a problem understanding what dataset to use when using cross validation for model evaluation. Let's say I have two models: LogisticRegression and ...
Michał Jurzak's user avatar
0 votes
1 answer
71 views

eval_metric of XGBoost // ML model in general

Say I am using Xgboost on a binary classification task. eval_metric is one of the model parameter. How should I think about the impact of using different eval_metric(e.g rmse/mae/logloss) in general? ...
pathtoagi's user avatar
0 votes
1 answer
205 views

How to compare test vs train model performance

When comparing the test vs train model performance to ensure no overfitting (e.g., using AUC ROC as an example), is it better to select the model with the largest test score, or the model with the ...
thereandhere1's user avatar
-1 votes
1 answer
123 views

Falcon 7B LLM Evaluation using TruLens

The problem I am facing is after defining the prompt template, creating a chain using Langchain, defining the huggingface evaluation module from trulens_eval to check the toxicity of the response and ...
RAUNAK GHOSH's user avatar
0 votes
0 answers
15 views

How do you figure out how much of a user rating to attribute to each step?

I've got a multi-step pipeline that produces output which is then (sometimes) rated by users. Something like this: Run sentiment analysis on input Run intent analysis on input Choose gating weights ...
TheEnvironmentalist's user avatar
-1 votes
1 answer
64 views

Which is the best binary classification model? Train and Test Accuracy are similar

I am building a binary classification model where classes are imbalanced but used SMOTE, I used 4 different models to compare performance and decide which to choose. They have same train and test ...
Sarah's user avatar
  • 1
0 votes
0 answers
13 views

One class confusion matrix notation for model evaluation

A one class classification set-up for a set of rules acting as a model, where each input is a whole dataset model makes some decision within the dataset for each entry output is decisions made for ...
reyna's user avatar
  • 101
0 votes
1 answer
32 views

model.evaluate gives low results?

i have an image dataset and there are 6300 images with 5 classes . The features extracted and dataset reduced to 256 features. This dataset gives good results(%99) when tested ANN with Backpropagation(...
ömer özcan's user avatar
0 votes
1 answer
233 views

Can I use GridSearchCV.best_score_ for evaluation of model performance?

Scikit-learn page on Grid Search says: Model selection by evaluating various parameter settings can be seen as a way to use the labeled data to “train” the parameters of the grid. When evaluating the ...
Charlie's user avatar
  • 103
0 votes
0 answers
17 views

Validation error less than training error

entire project link: Github Repository In a classification task using Neural Network, I computed the fraction of misclassification as an error. And I am getting a validation error less than the ...
Dipen Pandit's user avatar
2 votes
2 answers
512 views

Why do residuals of linear regression model need to be normally distributed?

When evaluating the output from a linear/ridge regression model, I have taken the residuals between the predicted and test data. This gives me a normal distribution when I plot this data as a ...
amy_hislop's user avatar
0 votes
1 answer
98 views

Evaluating overfitting in a logistic regression model

I have developed a logistic regression model for a classification problem and obtained an AUC (Area Under the Curve) score of approximately 0.9. The model was estimated by splitting the available data ...
Derrick's user avatar
0 votes
1 answer
25 views

How to evaluate machine translations of long documents?

I'm using Python and I want to compare the output of two machine-translation (ish) systems. Most of the tools seem to be focused on sentence-by-sentence evaluation. Either I get memory blow-ups with ...
Paul Prescod's user avatar
0 votes
0 answers
38 views

Which loss function should I use if multiple values are correct?

My task is to create a QA-model. I give it a context and a question that it should answer. The answer is usually one word, so a very simplified input would be e.g. Context: "Max eats a banana. ...
max245905's user avatar
0 votes
0 answers
25 views

Comparing Models based on two different sample sets of a single data set

I want to compare the performance of two different ML models like M1 and M2. I have a very huge data set and having two different downsampling of this data set, call them S1 and S2. Can I compare the ...
Vahid Shams's user avatar
0 votes
0 answers
20 views

torchmetrics BinaryMatthewsCorrCoef outputs 0 if target and prediction contains only one case either positive or negative case

I stared using MCC(Matthew's correlation coefficient) metric. But getting unexpected values when the given target and pred contains only one case either positive or negative (case - 1), The output of ...
lokesh's user avatar
  • 1
0 votes
1 answer
1k views

How to correctly measure the inference time and FLOPs of a model?

For some reason, I can’t find built-in solutions (not really?) in keras and tensorflow, while on the site https://keras.io/api/applications/ they provide Time (ms) per inference step (CPU), but for ...
Shadow_fiend's user avatar
1 vote
1 answer
33 views

Is there such thing as dataset imrovement?

I know that we can use explained machine learning to find why a model chose a certain classification. I wonder if there is a way I can find which features are going to improve my current model. I will ...
asmgx's user avatar
  • 549
1 vote
1 answer
118 views

Benchmarking LLMs on technical questions

There are several existing benchmark sets to evaluate the performance of large language models on natural-language comprehension tasks, such as CoQA, LAMBDA, HELLASWAG, LogiQA. I'm interested in ...
rwallace's user avatar
  • 149
0 votes
1 answer
80 views

How do I know If my regression model is underfitting?

How do we evaluate the performance of a regression model with a certain RMSE given that a domain knowledge performance metric is not present? Maybe MAPE is one way of comparing the performance of my ...
Mehmet Deniz's user avatar
0 votes
0 answers
19 views

What are some approaches to improve the classification of ONE particular instance of interest?

I'd like to know if there are some methods to correct a specific misclassified instance of interest (e.g. in a Logistic Regression or Random Forest). Like maybe increasing the error for that ...
Metrician's user avatar

1
2 3 4 5
8