Questions tagged [model-evaluations]
This tag is meant to be used for questions related to how to evaluate a model performance, not only based on standard metrics, but also in the context of real use case applications. What is a good model might depend on many factors to take into account, to eventually get really useful data science applications.
364
questions
0
votes
1
answer
113
views
Understand and compute confidence interval and coefficient of variation for regression model
I would like to better understand the concepts of:
coefficient of variation and confidence interval.
Trivially taking the definitions from wikipedia:
confidence interval (CI)
In frequentist ...
- 1
0
votes
1
answer
21
views
One class classification using Exclusively Positive Examples
I have a dataset consisting only of the positive class, and I want to train a model to identify this data. Is it possible to use one-class classification models for this task? Additionally, how can I ...
0
votes
0
answers
43
views
Actual values vs Preditec values plot
I was working on a project and I got a 0.98 R^2 score on both the training and test data sets and 0.91 training mse and 1.02 test mse, But my Actual values vs Predicted values looks like this, I was ...
0
votes
0
answers
15
views
Comparing a classfiers performance across distinct testing datasets
I have a dataset split into training, validation, and testing sets. I trained this model on the training data and evaluated on the validation and testing sets. Now I have an additional set of data ...
- 63
0
votes
0
answers
15
views
How to choose the correct NN model if the metrics are different in training and test time?
I am trying to build an LSTM model which has a lot of Dropout and Batch Norm Layers. When I run model.fit, the accuracy comes out to around 0.7 on the training data....
0
votes
0
answers
29
views
Does MAPE really penalizes overpredictions?
I read it on many sites, that one of the main "disadvantage" of MAPE is that it penalizes overpredictions, hence it prefers models that are under-predicting.
The main argument is that if we ...
- 25
0
votes
0
answers
11
views
Interpreting large discrepancies between Specificities & the # of Extraneous Variable Models selected by a variable selection algorithm
I am going to preface my question by saying that this problem of interpretation I have run into is in the context of me doing my part as a collaborator on a statistical learning paper for the first ...
- 157
0
votes
0
answers
15
views
Should I use an intercept even if my regression model's r-squared value reduces by a lot?
I'm using Python to create a good linear regression model and am having trouble getting good results for my r-squared value.
A quick rundown of what the data is:
– Sales: This dependent variable ...
0
votes
0
answers
22
views
Model Performance not improving
I am currently working with a GNN (a Graph attention Model) based model and the main task is to do Graph prediction. My model doesnot improve its performance when I change the number of heads or the ...
- 1
1
vote
2
answers
82
views
Xgboost model predicting extreme values for events and non-events | Overfitting
Extreme values are predicted by my trained xgboost classification model in BQML for both events (Y=1) and non-events (Y=0). For all event observations, the model calculates probability scores that ...
- 133
0
votes
0
answers
21
views
XGBoost Classifier Evaluation Confusion on New Dataset Despite High Cross-Validation Scores
I have built an XGBoost classifier model with 90 features, trained on a dataset containing 760k samples. I took great care to separate the labels from the features in both the training and testing ...
- 1
0
votes
1
answer
30
views
Different accuracy scores with sklearn roc_auc_score on same model using sklearn.metrics
Why do these below lines give different outputs while the input is the same?
I need to report these results in paper, but I am unsure which is better and why.
...
- 101
0
votes
1
answer
40
views
how to evaluate a model on our data when the model is imported from a library and thus not trained by us?
The company I work for has deployed a trained rule-based sentiment analyzer model vader to make predictions on customer's attitude. We import the model from nltk library directly, so we didn't train ...
- 3
2
votes
1
answer
94
views
Implementation of spBLEU
I was looking for a way to explore evaluation metrics for language translation models and I came across spBLEU. I can’t find any implementations/examples that would help me start. Does anyone have a ...
- 33
0
votes
0
answers
10
views
Is there a way to artificially manipulate a dataset in order to replace it for one that gives good predictions?
I'm trying to artificially create a dataset for pure educative reasons but I want it to be based in one particular dataset, the problem is that this original dataset don't make good predictions even ...
0
votes
3
answers
109
views
For cross validation should I use training set, or whole dataset?
I'm new to data science and I have a problem understanding what dataset to use when using cross validation for model evaluation.
Let's say I have two models: LogisticRegression and ...
0
votes
1
answer
45
views
eval_metric of XGBoost // ML model in general
Say I am using Xgboost on a binary classification task. eval_metric is one of the model parameter. How should I think about the impact of using different eval_metric(e.g rmse/mae/logloss) in general? ...
- 11
0
votes
1
answer
32
views
How to compare test vs train model performance
When comparing the test vs train model performance to ensure no overfitting (e.g., using AUC ROC as an example), is it better to select the model with the largest test score, or the model with the ...
- 735
-1
votes
1
answer
84
views
Falcon 7B LLM Evaluation using TruLens
The problem I am facing is after defining the prompt template, creating a chain using Langchain, defining the huggingface evaluation module from trulens_eval to check the toxicity of the response and ...
0
votes
0
answers
15
views
How do you figure out how much of a user rating to attribute to each step?
I've got a multi-step pipeline that produces output which is then (sometimes) rated by users. Something like this:
Run sentiment analysis on input
Run intent analysis on input
Choose gating weights ...
-1
votes
1
answer
33
views
Which is the best binary classification model? Train and Test Accuracy are similar
I am building a binary classification model where classes are imbalanced but used SMOTE, I used 4 different models to compare performance and decide which to choose.
They have same train and test ...
- 1
0
votes
0
answers
12
views
One class confusion matrix notation for model evaluation
A one class classification set-up for a set of rules acting as a model, where
each input is a whole dataset
model makes some decision within the dataset for each entry
output is decisions made for ...
- 101
0
votes
1
answer
29
views
model.evaluate gives low results?
i have an image dataset and there are 6300 images with 5 classes . The features extracted and dataset reduced to 256 features. This dataset gives good results(%99) when tested ANN with Backpropagation(...
0
votes
1
answer
72
views
Can I use GridSearchCV.best_score_ for evaluation of model performance?
Scikit-learn page on Grid Search says:
Model selection by evaluating various parameter settings can be seen
as a way to use the labeled data to “train” the parameters of the
grid.
When evaluating the ...
- 103
0
votes
0
answers
9
views
Validation error less than training error
entire project link: Github Repository
In a classification task using Neural Network, I computed the fraction of misclassification as an error. And I am getting a validation error less than the ...
2
votes
2
answers
142
views
Why do residuals of linear regression model need to be normally distributed?
When evaluating the output from a linear/ridge regression model, I have taken the residuals between the predicted and test data. This gives me a normal distribution when I plot this data as a ...
- 23
0
votes
1
answer
58
views
Evaluating overfitting in a logistic regression model
I have developed a logistic regression model for a classification problem and obtained an AUC (Area Under the Curve) score of approximately 0.9. The model was estimated by splitting the available data ...
- 1
0
votes
1
answer
23
views
How to evaluate machine translations of long documents?
I'm using Python and I want to compare the output of two machine-translation (ish) systems. Most of the tools seem to be focused on sentence-by-sentence evaluation. Either I get memory blow-ups with ...
- 101
0
votes
0
answers
29
views
Which loss function should I use if multiple values are correct?
My task is to create a QA-model. I give it a context and a question that it should answer. The answer is usually one word, so a very simplified input would be e.g.
Context: "Max eats a banana. ...
0
votes
0
answers
24
views
Comparing Models based on two different sample sets of a single data set
I want to compare the performance of two different ML models like M1 and M2. I have a very huge data set and having two different downsampling of this data set, call them S1 and S2. Can I compare the ...
- 113
0
votes
0
answers
19
views
torchmetrics BinaryMatthewsCorrCoef outputs 0 if target and prediction contains only one case either positive or negative case
I stared using MCC(Matthew's correlation coefficient) metric. But getting unexpected values when the given target and pred contains only one case either positive or negative (case - 1), The output of ...
- 1
0
votes
1
answer
660
views
How to correctly measure the inference time and FLOPs of a model?
For some reason, I can’t find built-in solutions (not really?) in keras and tensorflow, while on the site
https://keras.io/api/applications/ they provide Time (ms) per inference step (CPU), but for ...
1
vote
1
answer
29
views
Is there such thing as dataset imrovement?
I know that we can use explained machine learning to find why a model chose a certain classification.
I wonder if there is a way I can find which features are going to improve my current model.
I will ...
- 539
0
votes
0
answers
178
views
Multilabel metrics: micro vs. macro vs. weighted vs. samples?
I'm working on a multilabel classification problem; there are $N$ classes and each example can belong to $[0, N]$ of those classes. Below you can see the precision and recall computed using various ...
0
votes
0
answers
13
views
why performance values for passive aggressive model changing every time?
i havq question, that am a little confused about PassiveAggressive performance values after i fitted tf-idf vectorization on it(bigram and trigram), the values are changed every time i run the code ...
- 1
1
vote
1
answer
95
views
Benchmarking LLMs on technical questions
There are several existing benchmark sets to evaluate the performance of large language models on natural-language comprehension tasks, such as CoQA, LAMBDA, HELLASWAG, LogiQA.
I'm interested in ...
- 139
0
votes
0
answers
8
views
What are some commonly used non-threshold dependent classification predictive performance metrics?
I'm working on a prediction framework that utilizes a classification algorithm, trained on data with a binary outcome, and makes probabilistic predictions. I'm looking for some help determining ...
- 1
0
votes
0
answers
22
views
Should I specify 'reference=train_set' when creating a evaluating lightgbm.Dataset?
According to the docs, we should offer a argument reference=train_set when creating a validating lightgbm.Dataset.
I know a little about the reason, i.e. there are ...
- 101
0
votes
0
answers
26
views
SemEval-2016 Task 5 Subtask 2 Evaluation
Hi, I read SEMEVAL task 5 from this website (https://alt.qcri.org/semeval2016/task5/) but the evaluation method for subtask 2 is not clear. I really appreciate it if you tell me the method. This is ...
0
votes
1
answer
61
views
How do I know If my regression model is underfitting?
How do we evaluate the performance of a regression model with a certain RMSE given that a domain knowledge performance metric is not present?
Maybe MAPE is one way of comparing the performance of my ...
- 31
0
votes
0
answers
27
views
Workflow for improving and comparing deep learning models
Say I have the most basic neural network that performs some task (eg Keras sequential model with one hidden layer, used to binary classification) and a list of ideas how one could improve it (like: ...
- 103
0
votes
0
answers
18
views
What are some approaches to improve the classification of ONE particular instance of interest?
I'd like to know if there are some methods to correct a specific misclassified instance of interest (e.g. in a Logistic Regression or Random Forest).
Like maybe increasing the error for that ...
- 11
1
vote
1
answer
58
views
Imbalanced performance metrics in binary classification
I am developing a binary classification model using sklearn pipeline for preprocessing and a soft voting classifier (Adaboost and Extratrees with 50 estimators). The dataset (3 million rows) contains ...
- 23
2
votes
1
answer
1k
views
How do we evaluate the outputs of text generation models?
Evaluation of a wide variety of natural language generation (NLG) tasks is difficult. For instance, for a question answering model, it is hard for a human to quantify how well the model has answered a ...
- 121
0
votes
0
answers
18
views
Kernel Density Estimation and performance evaluation
I am doing a data science project about Kernel Density Estimation, specifically about finding the best bandwidth and kernel function to use.
I need to use data that I don't know the actual underlying ...
0
votes
1
answer
128
views
Why is the accuracy on train dataset not always 100% while we use the same dataset to train the model?
Though tree-based ML algorithms give us 100% accuracy on train dataset many times, but why is this not happening every time. I know this results in overfitting but why not 100% accuracy every time on ...
0
votes
0
answers
38
views
Large change in validation loss, small change in training loss
I'm training a multi-task, multi-label neural network. I am attempting to tune the architecture and am having some trouble interpreting the learning curves. Particularly, when I look at the learning ...
- 63
2
votes
1
answer
89
views
How to present a statistical justification for the choice of models with approximate accuracies?
In an experiment involving the comparison of classification algorithms, how can I assess whether there are statistically significant differences between the analyzed models? For example, the following ...
0
votes
0
answers
15
views
Evaluating models which classify on rolling time intervals
TLDR: I am trying to predict the probability of an incident occurring within a specific time interval. I have data from multiple years, and I know the exact time of year that incidents occur. I have ...
- 1
0
votes
1
answer
97
views
Export a detailed report with metrics after the training process of a Keras model
I have an image-segmentation model (.h5) which was trained using dice coefficient, recall and precision metrics.
The model was trained using the ImageDataGenerator class with the following code:
...
- 111