Skip to main content

Questions tagged [model-evaluations]

This tag is meant to be used for questions related to how to evaluate a model performance, not only based on standard metrics, but also in the context of real use case applications. What is a good model might depend on many factors to take into account, to eventually get really useful data science applications.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

What's a suitable error metric for quantifying uncertainty in future projections using a model ensemble?

Rather than just providing a mean projection, I'm looking to provide a likely range of projections using output from 9 models. Each dataset consists of spatial maximum probability values [0,1] for a ...
liveFreeOrπHard's user avatar
0 votes
1 answer
42 views

does this ROC curve and the ROC AUC score really match?

I am working on a binary classification problem. I tried to evaluate a model by plotting ROC curve and calculating ROC AUC score. The calculated score is 0.9115 but the curve area looks not releastic ...
MoeCaruso's user avatar
1 vote
0 answers
28 views

Evaluating isolation forest (Unsupervised learning use case)

When using Isolation Forest for outlier detection (an unsupervised case), how to evaluate the performance of isolation forest?
Ram's user avatar
  • 11
0 votes
0 answers
14 views

Impact of Adding Imbalanced Data on Model Performance for Different Groups

Suppose I initially have a dataset with 50 samples of type A and 50 samples of type B, each with several features. I built a neural network model using this data and recorded the prediction accuracy ...
Mickly's user avatar
  • 1
0 votes
0 answers
15 views

Why PySpark `BinaryClasssificationEvaluator` metric `areaUnderROC` returns slightly different across multiple evaluations on the same dataset?

I am using BinaryClasssificationEvaluator in Pyspark to calculate AUC, however, I find that the returned auc across multiple evaluations on the same dataset are ...
helloworld's user avatar
0 votes
0 answers
6 views

ways to calculate the accuracy of a object detection model

In case of classification, we will calculate the accuracy by performance of model on test dataset. ...
MohanGandhi's user avatar
0 votes
0 answers
23 views

Theoretical Limitations of Achieving 100% Accuracy in Modeling Non-linear Relationships with Neural Networks

I am working on a project where I need to model a specific non-linear relationship using a neural network. The relationship is given by $y = 3x_1^2x_2^3 $. The approach involves: Preprocessing the ...
Mo McWebmo's user avatar
1 vote
1 answer
55 views

How to measure different models' feature importance using a generic and common standard?

I want to measure the feature importance of a series of models after training them. Most models have some built-in APIs that allow me to access their feature importance, but as far as I know, these ...
Yuuya's user avatar
  • 31
0 votes
0 answers
18 views

How to use cross validation to select/evaluate model with probability score as the output?

Initially I was evaluating my models using cross_val with out-of-pocket metrics such as precision, recall, f1 score, etc, or with my own metrics defined in ...
szheng's user avatar
  • 21
1 vote
1 answer
105 views

How do I work with time-series data of temperature?

So I have some equipment temperature and i have outside temperature (both are collected daily) and I want to predict the equipment temperature. However, I'm new to this and unsure about which model to ...
Ria's user avatar
  • 11
0 votes
0 answers
31 views

What are the most important evaluation metrics for anomaly segmentation?

When people talk about anomaly segmentation models, they often mention evaluation metrics like F1 score, AP, AUROC, and AUPRO. But which one really matters most when comparing models, and why? I'm ...
Mosh Geb's user avatar
1 vote
1 answer
17 views

Reduce mode searching behaviour of VAE

I'm applying VAEs to sections genomic data (haplotypic vcf format, so binary variables), with one model being trained on each section. They each have different layer sizes and weights to better fit ...
Whitehot's user avatar
  • 121
0 votes
0 answers
9 views

İnternal and hold out test for shap

There is an internal test set and a hold out test set. I explain the model with Shap library. Should I use the internal test set or the external test set? What should I do if there is a difference?
Nemo's user avatar
  • 1
0 votes
0 answers
10 views

How to assess the stability of a DL model, after using k-fold cross-validation for hyperparameter tuning

I've recently completed the training of a deep learning model for a classification task, using a process that involves k-fold cross-validation for hyperparameter tuning Initially, I have divided my ...
o'hara's user avatar
  • 1
0 votes
0 answers
15 views

what is the main difference between ROUGE and BLUE?

Both (ROUGE, BLUE) are useful to find the similarity between machine generated summary and reference summary. what is the main difference?
Tovlk's user avatar
  • 43
0 votes
0 answers
37 views

Why is there a difference in Training Accuracy Output, when the training dataset is the same but the validation dataset is different?

I am looking at the output of a multi-class image segmentation deep learning model. I used U-Net to implement this. I am confused about why the training accuracies are different for a different ...
user10529827's user avatar
0 votes
0 answers
33 views

Difference in the value of evaluation-metric in xgb.train() and predict in R

I have trained a xgboost classifier with a custom metric (f1_xgb), that is, the F1 score. Here the important aspect is that I evaluated the classifier on the test set by setting: ...
user159339's user avatar
2 votes
1 answer
110 views

How to interpret RMSE to evaluate a regression model

I am trying to evaluate a regression model (random forests); my understanding is that R^2 (coefficient of determination) is not a good measure of fitness since my dataset is non-linear. It looks like ...
Shawn's user avatar
  • 35
0 votes
0 answers
29 views

How to treat "ignore" boxes during object detector evaluation?

I want to evaluate the performance of multiple object detectors for comparison, each of them has been trained on this data. This dataset has images labeled, where each image can have multiple labels. ...
hafiz031's user avatar
  • 121
1 vote
1 answer
27 views

Topic modeling evaluation

I'm working on topic modeling and I have generated clusters with two different methods. How can I evaluate which method performs better than the other?
user5520049's user avatar
0 votes
0 answers
39 views

PR AUC curve with drop in precision

I have this PR AUC plot, with both PCA and autoencoder related curves having a huge drop of precision in the beginning and then increasing again, with PCA hitting 0 as you can see in the zoomed in ...
GabrielPast's user avatar
0 votes
1 answer
165 views

Understand and compute confidence interval and coefficient of variation for regression model

I would like to better understand the concepts of: coefficient of variation and confidence interval. Trivially taking the definitions from wikipedia: confidence interval (CI) In frequentist ...
Cata's user avatar
  • 1
0 votes
1 answer
43 views

One class classification using Exclusively Positive Examples

I have a dataset consisting only of the positive class, and I want to train a model to identify this data. Is it possible to use one-class classification models for this task? Additionally, how can I ...
user13241829's user avatar
0 votes
0 answers
77 views

Actual values vs Preditec values plot

I was working on a project and I got a 0.98 R^2 score on both the training and test data sets and 0.91 training mse and 1.02 test mse, But my Actual values vs Predicted values looks like this, I was ...
Ario Alavi's user avatar
0 votes
0 answers
15 views

Comparing a classfiers performance across distinct testing datasets

I have a dataset split into training, validation, and testing sets. I trained this model on the training data and evaluated on the validation and testing sets. Now I have an additional set of data ...
tensormoby's user avatar
0 votes
0 answers
22 views

How to choose the correct NN model if the metrics are different in training and test time?

I am trying to build an LSTM model which has a lot of Dropout and Batch Norm Layers. When I run model.fit, the accuracy comes out to around 0.7 on the training data....
Jeffrey Davidson's user avatar
0 votes
0 answers
52 views

Does MAPE really penalizes overpredictions?

I read it on many sites, that one of the main "disadvantage" of MAPE is that it penalizes overpredictions, hence it prefers models that are under-predicting. The main argument is that if we ...
morqueatsz's user avatar
0 votes
0 answers
12 views

Interpreting large discrepancies between Specificities & the # of Extraneous Variable Models selected by a variable selection algorithm

I am going to preface my question by saying that this problem of interpretation I have run into is in the context of me doing my part as a collaborator on a statistical learning paper for the first ...
Marlen's user avatar
  • 167
0 votes
0 answers
19 views

Should I use an intercept even if my regression model's r-squared value reduces by a lot?

I'm using Python to create a good linear regression model and am having trouble getting good results for my r-squared value. A quick rundown of what the data is: – Sales: This dependent variable ...
Python Student's user avatar
0 votes
0 answers
23 views

Model Performance not improving

I am currently working with a GNN (a Graph attention Model) based model and the main task is to do Graph prediction. My model doesnot improve its performance when I change the number of heads or the ...
Susan's user avatar
  • 1
1 vote
2 answers
164 views

Xgboost model predicting extreme values for events and non-events | Overfitting

Extreme values are predicted by my trained xgboost classification model in BQML for both events (Y=1) and non-events (Y=0). For all event observations, the model calculates probability scores that ...
Grammilo's user avatar
  • 133
0 votes
0 answers
61 views

XGBoost Classifier Evaluation Confusion on New Dataset Despite High Cross-Validation Scores

I have built an XGBoost classifier model with 90 features, trained on a dataset containing 760k samples. I took great care to separate the labels from the features in both the training and testing ...
oklen's user avatar
  • 1
0 votes
1 answer
149 views

Different accuracy scores with sklearn roc_auc_score on same model using sklearn.metrics

Why do these below lines give different outputs while the input is the same? I need to report these results in paper, but I am unsure which is better and why. ...
Adnan Ali's user avatar
  • 101
0 votes
1 answer
41 views

how to evaluate a model on our data when the model is imported from a library and thus not trained by us?

The company I work for has deployed a trained rule-based sentiment analyzer model vader to make predictions on customer's attitude. We import the model from nltk library directly, so we didn't train ...
Shelby's user avatar
  • 3
2 votes
1 answer
410 views

Implementation of spBLEU

I was looking for a way to explore evaluation metrics for language translation models and I came across spBLEU. I can’t find any implementations/examples that would help me start. Does anyone have a ...
Prithvi's user avatar
  • 33
0 votes
0 answers
10 views

Is there a way to artificially manipulate a dataset in order to replace it for one that gives good predictions?

I'm trying to artificially create a dataset for pure educative reasons but I want it to be based in one particular dataset, the problem is that this original dataset don't make good predictions even ...
CinfaCinfa's user avatar
0 votes
3 answers
451 views

For cross validation should I use training set, or whole dataset?

I'm new to data science and I have a problem understanding what dataset to use when using cross validation for model evaluation. Let's say I have two models: LogisticRegression and ...
Michał Jurzak's user avatar
0 votes
1 answer
74 views

eval_metric of XGBoost // ML model in general

Say I am using Xgboost on a binary classification task. eval_metric is one of the model parameter. How should I think about the impact of using different eval_metric(e.g rmse/mae/logloss) in general? ...
pathtoagi's user avatar
0 votes
1 answer
315 views

How to compare test vs train model performance

When comparing the test vs train model performance to ensure no overfitting (e.g., using AUC ROC as an example), is it better to select the model with the largest test score, or the model with the ...
thereandhere1's user avatar
-1 votes
1 answer
127 views

Falcon 7B LLM Evaluation using TruLens

The problem I am facing is after defining the prompt template, creating a chain using Langchain, defining the huggingface evaluation module from trulens_eval to check the toxicity of the response and ...
RAUNAK GHOSH's user avatar
0 votes
0 answers
15 views

How do you figure out how much of a user rating to attribute to each step?

I've got a multi-step pipeline that produces output which is then (sometimes) rated by users. Something like this: Run sentiment analysis on input Run intent analysis on input Choose gating weights ...
TheEnvironmentalist's user avatar
-1 votes
1 answer
124 views

Which is the best binary classification model? Train and Test Accuracy are similar

I am building a binary classification model where classes are imbalanced but used SMOTE, I used 4 different models to compare performance and decide which to choose. They have same train and test ...
Sarah's user avatar
  • 1
0 votes
1 answer
32 views

model.evaluate gives low results?

i have an image dataset and there are 6300 images with 5 classes . The features extracted and dataset reduced to 256 features. This dataset gives good results(%99) when tested ANN with Backpropagation(...
ömer özcan's user avatar
0 votes
1 answer
357 views

Can I use GridSearchCV.best_score_ for evaluation of model performance?

Scikit-learn page on Grid Search says: Model selection by evaluating various parameter settings can be seen as a way to use the labeled data to “train” the parameters of the grid. When evaluating the ...
Charlie's user avatar
  • 103
2 votes
2 answers
711 views

Why do residuals of linear regression model need to be normally distributed?

When evaluating the output from a linear/ridge regression model, I have taken the residuals between the predicted and test data. This gives me a normal distribution when I plot this data as a ...
amy_hislop's user avatar
0 votes
1 answer
119 views

Evaluating overfitting in a logistic regression model

I have developed a logistic regression model for a classification problem and obtained an AUC (Area Under the Curve) score of approximately 0.9. The model was estimated by splitting the available data ...
Derrick's user avatar
0 votes
1 answer
26 views

How to evaluate machine translations of long documents?

I'm using Python and I want to compare the output of two machine-translation (ish) systems. Most of the tools seem to be focused on sentence-by-sentence evaluation. Either I get memory blow-ups with ...
Paul Prescod's user avatar
0 votes
1 answer
1k views

How to correctly measure the inference time and FLOPs of a model?

For some reason, I can’t find built-in solutions (not really?) in keras and tensorflow, while on the site https://keras.io/api/applications/ they provide Time (ms) per inference step (CPU), but for ...
Shadow_fiend's user avatar
1 vote
1 answer
33 views

Is there such thing as dataset imrovement?

I know that we can use explained machine learning to find why a model chose a certain classification. I wonder if there is a way I can find which features are going to improve my current model. I will ...
asmgx's user avatar
  • 549
1 vote
1 answer
132 views

Benchmarking LLMs on technical questions

There are several existing benchmark sets to evaluate the performance of large language models on natural-language comprehension tasks, such as CoQA, LAMBDA, HELLASWAG, LogiQA. I'm interested in ...
rwallace's user avatar
  • 149

1
2 3 4 5
8