Questions tagged [model-evaluations]

This tag is meant to be used for questions related to how to evaluate a model performance, not only based on standard metrics, but also in the context of real use case applications. What is a good model might depend on many factors to take into account, to eventually get really useful data science applications.

Filter by
Sorted by
Tagged with
0 votes
1 answer
113 views

Understand and compute confidence interval and coefficient of variation for regression model

I would like to better understand the concepts of: coefficient of variation and confidence interval. Trivially taking the definitions from wikipedia: confidence interval (CI) In frequentist ...
Cata's user avatar
  • 1
0 votes
1 answer
21 views

One class classification using Exclusively Positive Examples

I have a dataset consisting only of the positive class, and I want to train a model to identify this data. Is it possible to use one-class classification models for this task? Additionally, how can I ...
user13241829's user avatar
0 votes
0 answers
43 views

Actual values vs Preditec values plot

I was working on a project and I got a 0.98 R^2 score on both the training and test data sets and 0.91 training mse and 1.02 test mse, But my Actual values vs Predicted values looks like this, I was ...
Ario Alavi's user avatar
0 votes
0 answers
15 views

Comparing a classfiers performance across distinct testing datasets

I have a dataset split into training, validation, and testing sets. I trained this model on the training data and evaluated on the validation and testing sets. Now I have an additional set of data ...
tensormoby's user avatar
0 votes
0 answers
15 views

How to choose the correct NN model if the metrics are different in training and test time?

I am trying to build an LSTM model which has a lot of Dropout and Batch Norm Layers. When I run model.fit, the accuracy comes out to around 0.7 on the training data....
Jeffrey Davidson's user avatar
0 votes
0 answers
29 views

Does MAPE really penalizes overpredictions?

I read it on many sites, that one of the main "disadvantage" of MAPE is that it penalizes overpredictions, hence it prefers models that are under-predicting. The main argument is that if we ...
morqueatsz's user avatar
0 votes
0 answers
11 views

Interpreting large discrepancies between Specificities & the # of Extraneous Variable Models selected by a variable selection algorithm

I am going to preface my question by saying that this problem of interpretation I have run into is in the context of me doing my part as a collaborator on a statistical learning paper for the first ...
Marlen's user avatar
  • 157
0 votes
0 answers
15 views

Should I use an intercept even if my regression model's r-squared value reduces by a lot?

I'm using Python to create a good linear regression model and am having trouble getting good results for my r-squared value. A quick rundown of what the data is: – Sales: This dependent variable ...
Python Student's user avatar
0 votes
0 answers
22 views

Model Performance not improving

I am currently working with a GNN (a Graph attention Model) based model and the main task is to do Graph prediction. My model doesnot improve its performance when I change the number of heads or the ...
Susan's user avatar
  • 1
1 vote
2 answers
82 views

Xgboost model predicting extreme values for events and non-events | Overfitting

Extreme values are predicted by my trained xgboost classification model in BQML for both events (Y=1) and non-events (Y=0). For all event observations, the model calculates probability scores that ...
Scott Grammilo's user avatar
0 votes
0 answers
21 views

XGBoost Classifier Evaluation Confusion on New Dataset Despite High Cross-Validation Scores

I have built an XGBoost classifier model with 90 features, trained on a dataset containing 760k samples. I took great care to separate the labels from the features in both the training and testing ...
oklen's user avatar
  • 1
0 votes
1 answer
30 views

Different accuracy scores with sklearn roc_auc_score on same model using sklearn.metrics

Why do these below lines give different outputs while the input is the same? I need to report these results in paper, but I am unsure which is better and why. ...
Adnan Ali's user avatar
  • 101
0 votes
1 answer
40 views

how to evaluate a model on our data when the model is imported from a library and thus not trained by us?

The company I work for has deployed a trained rule-based sentiment analyzer model vader to make predictions on customer's attitude. We import the model from nltk library directly, so we didn't train ...
Shelby's user avatar
  • 3
2 votes
1 answer
94 views

Implementation of spBLEU

I was looking for a way to explore evaluation metrics for language translation models and I came across spBLEU. I can’t find any implementations/examples that would help me start. Does anyone have a ...
Prithvi's user avatar
  • 33
0 votes
0 answers
10 views

Is there a way to artificially manipulate a dataset in order to replace it for one that gives good predictions?

I'm trying to artificially create a dataset for pure educative reasons but I want it to be based in one particular dataset, the problem is that this original dataset don't make good predictions even ...
CinfaCinfa's user avatar
0 votes
3 answers
109 views

For cross validation should I use training set, or whole dataset?

I'm new to data science and I have a problem understanding what dataset to use when using cross validation for model evaluation. Let's say I have two models: LogisticRegression and ...
Michał Jurzak's user avatar
0 votes
1 answer
45 views

eval_metric of XGBoost // ML model in general

Say I am using Xgboost on a binary classification task. eval_metric is one of the model parameter. How should I think about the impact of using different eval_metric(e.g rmse/mae/logloss) in general? ...
pathtoagi's user avatar
0 votes
1 answer
32 views

How to compare test vs train model performance

When comparing the test vs train model performance to ensure no overfitting (e.g., using AUC ROC as an example), is it better to select the model with the largest test score, or the model with the ...
thereandhere1's user avatar
-1 votes
1 answer
84 views

Falcon 7B LLM Evaluation using TruLens

The problem I am facing is after defining the prompt template, creating a chain using Langchain, defining the huggingface evaluation module from trulens_eval to check the toxicity of the response and ...
RAUNAK GHOSH's user avatar
0 votes
0 answers
15 views

How do you figure out how much of a user rating to attribute to each step?

I've got a multi-step pipeline that produces output which is then (sometimes) rated by users. Something like this: Run sentiment analysis on input Run intent analysis on input Choose gating weights ...
TheEnvironmentalist's user avatar
-1 votes
1 answer
33 views

Which is the best binary classification model? Train and Test Accuracy are similar

I am building a binary classification model where classes are imbalanced but used SMOTE, I used 4 different models to compare performance and decide which to choose. They have same train and test ...
Sarah's user avatar
  • 1
0 votes
0 answers
12 views

One class confusion matrix notation for model evaluation

A one class classification set-up for a set of rules acting as a model, where each input is a whole dataset model makes some decision within the dataset for each entry output is decisions made for ...
zaira's user avatar
  • 101
0 votes
1 answer
29 views

model.evaluate gives low results?

i have an image dataset and there are 6300 images with 5 classes . The features extracted and dataset reduced to 256 features. This dataset gives good results(%99) when tested ANN with Backpropagation(...
ömer özcan's user avatar
0 votes
1 answer
72 views

Can I use GridSearchCV.best_score_ for evaluation of model performance?

Scikit-learn page on Grid Search says: Model selection by evaluating various parameter settings can be seen as a way to use the labeled data to “train” the parameters of the grid. When evaluating the ...
Charlie's user avatar
  • 103
0 votes
0 answers
9 views

Validation error less than training error

entire project link: Github Repository In a classification task using Neural Network, I computed the fraction of misclassification as an error. And I am getting a validation error less than the ...
Dipen Pandit's user avatar
2 votes
2 answers
142 views

Why do residuals of linear regression model need to be normally distributed?

When evaluating the output from a linear/ridge regression model, I have taken the residuals between the predicted and test data. This gives me a normal distribution when I plot this data as a ...
amy_hislop's user avatar
0 votes
1 answer
58 views

Evaluating overfitting in a logistic regression model

I have developed a logistic regression model for a classification problem and obtained an AUC (Area Under the Curve) score of approximately 0.9. The model was estimated by splitting the available data ...
Derrick's user avatar
0 votes
1 answer
23 views

How to evaluate machine translations of long documents?

I'm using Python and I want to compare the output of two machine-translation (ish) systems. Most of the tools seem to be focused on sentence-by-sentence evaluation. Either I get memory blow-ups with ...
Paul Prescod's user avatar
0 votes
0 answers
29 views

Which loss function should I use if multiple values are correct?

My task is to create a QA-model. I give it a context and a question that it should answer. The answer is usually one word, so a very simplified input would be e.g. Context: "Max eats a banana. ...
max245905's user avatar
0 votes
0 answers
24 views

Comparing Models based on two different sample sets of a single data set

I want to compare the performance of two different ML models like M1 and M2. I have a very huge data set and having two different downsampling of this data set, call them S1 and S2. Can I compare the ...
Vahid Shams's user avatar
0 votes
0 answers
19 views

torchmetrics BinaryMatthewsCorrCoef outputs 0 if target and prediction contains only one case either positive or negative case

I stared using MCC(Matthew's correlation coefficient) metric. But getting unexpected values when the given target and pred contains only one case either positive or negative (case - 1), The output of ...
lokesh's user avatar
  • 1
0 votes
1 answer
660 views

How to correctly measure the inference time and FLOPs of a model?

For some reason, I can’t find built-in solutions (not really?) in keras and tensorflow, while on the site https://keras.io/api/applications/ they provide Time (ms) per inference step (CPU), but for ...
Shadow_fiend's user avatar
1 vote
1 answer
29 views

Is there such thing as dataset imrovement?

I know that we can use explained machine learning to find why a model chose a certain classification. I wonder if there is a way I can find which features are going to improve my current model. I will ...
asmgx's user avatar
  • 539
0 votes
0 answers
178 views

Multilabel metrics: micro vs. macro vs. weighted vs. samples?

I'm working on a multilabel classification problem; there are $N$ classes and each example can belong to $[0, N]$ of those classes. Below you can see the precision and recall computed using various ...
Each One Chew's user avatar
0 votes
0 answers
13 views

why performance values for passive aggressive model changing every time?

i havq question, that am a little confused about PassiveAggressive performance values after i fitted tf-idf vectorization on it(bigram and trigram), the values are changed every time i run the code ...
joj abd's user avatar
1 vote
1 answer
95 views

Benchmarking LLMs on technical questions

There are several existing benchmark sets to evaluate the performance of large language models on natural-language comprehension tasks, such as CoQA, LAMBDA, HELLASWAG, LogiQA. I'm interested in ...
rwallace's user avatar
  • 139
0 votes
0 answers
8 views

What are some commonly used non-threshold dependent classification predictive performance metrics?

I'm working on a prediction framework that utilizes a classification algorithm, trained on data with a binary outcome, and makes probabilistic predictions. I'm looking for some help determining ...
Ted's user avatar
  • 1
0 votes
0 answers
22 views

Should I specify 'reference=train_set' when creating a evaluating lightgbm.Dataset?

According to the docs, we should offer a argument reference=train_set when creating a validating lightgbm.Dataset. I know a little about the reason, i.e. there are ...
EvilRoach's user avatar
  • 101
0 votes
0 answers
26 views

SemEval-2016 Task 5 Subtask 2 Evaluation

Hi, I read SEMEVAL task 5 from this website (https://alt.qcri.org/semeval2016/task5/) but the evaluation method for subtask 2 is not clear. I really appreciate it if you tell me the method. This is ...
mansoor sh's user avatar
0 votes
1 answer
61 views

How do I know If my regression model is underfitting?

How do we evaluate the performance of a regression model with a certain RMSE given that a domain knowledge performance metric is not present? Maybe MAPE is one way of comparing the performance of my ...
Mehmet Deniz's user avatar
0 votes
0 answers
27 views

Workflow for improving and comparing deep learning models

Say I have the most basic neural network that performs some task (eg Keras sequential model with one hidden layer, used to binary classification) and a list of ideas how one could improve it (like: ...
Brzoskwinia's user avatar
0 votes
0 answers
18 views

What are some approaches to improve the classification of ONE particular instance of interest?

I'd like to know if there are some methods to correct a specific misclassified instance of interest (e.g. in a Logistic Regression or Random Forest). Like maybe increasing the error for that ...
Metrician's user avatar
1 vote
1 answer
58 views

Imbalanced performance metrics in binary classification

I am developing a binary classification model using sklearn pipeline for preprocessing and a soft voting classifier (Adaboost and Extratrees with 50 estimators). The dataset (3 million rows) contains ...
fendrbud's user avatar
2 votes
1 answer
1k views

How do we evaluate the outputs of text generation models?

Evaluation of a wide variety of natural language generation (NLG) tasks is difficult. For instance, for a question answering model, it is hard for a human to quantify how well the model has answered a ...
Greggs's user avatar
  • 121
0 votes
0 answers
18 views

Kernel Density Estimation and performance evaluation

I am doing a data science project about Kernel Density Estimation, specifically about finding the best bandwidth and kernel function to use. I need to use data that I don't know the actual underlying ...
Liel Gutman's user avatar
0 votes
1 answer
128 views

Why is the accuracy on train dataset not always 100% while we use the same dataset to train the model?

Though tree-based ML algorithms give us 100% accuracy on train dataset many times, but why is this not happening every time. I know this results in overfitting but why not 100% accuracy every time on ...
Mystical_soul's user avatar
0 votes
0 answers
38 views

Large change in validation loss, small change in training loss

I'm training a multi-task, multi-label neural network. I am attempting to tune the architecture and am having some trouble interpreting the learning curves. Particularly, when I look at the learning ...
tensormoby's user avatar
2 votes
1 answer
89 views

How to present a statistical justification for the choice of models with approximate accuracies?

In an experiment involving the comparison of classification algorithms, how can I assess whether there are statistically significant differences between the analyzed models? For example, the following ...
Paulo Sergio Moreira's user avatar
0 votes
0 answers
15 views

Evaluating models which classify on rolling time intervals

TLDR: I am trying to predict the probability of an incident occurring within a specific time interval. I have data from multiple years, and I know the exact time of year that incidents occur. I have ...
zsakib_'s user avatar
0 votes
1 answer
97 views

Export a detailed report with metrics after the training process of a Keras model

I have an image-segmentation model (.h5) which was trained using dice coefficient, recall and precision metrics. The model was trained using the ImageDataGenerator class with the following code: ...
Capdi's user avatar
  • 111

1
2 3 4 5
8