Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

Filter by
Sorted by
Tagged with
1
vote
0answers
10 views

What are the business metrics I should track to evaluate a recommender model deployed on an e-commerce website? [closed]

Can you suggest some google analytics metrics such as (click or impressions etc) to evaluate a recommender model deployed on an e-commerce website.
1
vote
0answers
19 views

(Graph Convolutional Network (GCN) based recommender system maintenance issue [closed]

I have built an item-item recommender model using (Graph Convolutional Network (GCN) for an E-commerce website. Could you please help me with the maintenance of the model. How often should I retrain ...
1
vote
0answers
9 views

What are the motion planning benchmarks?

Suppose I wanted to try and improve on existing motion planning algorithms. What benchmarks should I be trying to beat? Papers with code site has no motion planning benchmarks. I searched online and ...
2
votes
1answer
15 views

Output of evaluation metric for XGBoost - is it cumulative?

On the 10th boosting round for XGBoost, I get an MAP of 0.32 on the test data. Does that reflect the performance of just that 10th tree? Or the performance of all 10 trees combined that have been ...
0
votes
0answers
73 views

Shouldn't ROUGE-1 precision be equal to BLEU with w=(1, 0, 0, 0) when brevity penalty is 1?

I am trying to evaluate a NLP model using BLEU and ROUGE. However, I am a bit confused about the difference between those scores. While I am aware that ROUGE is aimed at recall whilst BLEU measures ...
1
vote
1answer
20 views

Evaluation of recommendation systems

I have developed a content-based recommendation system and it is working fine. The input is a set of documents={d1,d2,d3,...,dn} and the output will be Top N similar documents for a given document ...
3
votes
1answer
31 views

Evaluation metric for Information retrieval system

I am currently reading Semantic Product Search paper published by Amazon. They are using two evaluation subtasks matching and ranking. In matching, they tune the model hyperparameters to maximize ...
0
votes
0answers
34 views

How can realize the evaluation/validation of unsupervised models through unlabeled data?

I'm researching anomaly detection, which is nothing else than outliers detection on a set of time-series web servers access log data or network traffic. Recently I re-faced to following fundamental ...
1
vote
1answer
50 views

Micro Average vs Macro Average for Class Imbalance

I have a dataset consisting of around 30'000 data points and 3 classes. The classes are imbalanced (around 5'000 in class 1, 10'000 in class 2 and 15'000 in class 3). I'm building a convolutional ...
0
votes
0answers
17 views

Using accuracy metric during training for unbalanced multiclass classification

I am training a convolutional neural network and the sensitivity and precision of the minority class is what is most important to me. I am using 10-Fold cross validation, and the test fold is ...
0
votes
1answer
31 views

How to estimate missing values when calculating NDCG

I would like to compare recommendations methods using NDCG metric on MovieLens dataset. In ranking problem, the goal is to rank items based on their relevance for user. Ranking models can be learned ...
1
vote
1answer
79 views

Macro and micro average for imbalanced binary classes

Micro and macro averaging are metrics for multi-class classification. However, for binary classification when data are imbalanced, it seems that micro and macro precision have different results. My ...
1
vote
1answer
54 views

Proper evaluation method for recommendation system with implicit feedback?

I am trying to implement a recommendation system for a live-streaming website. Here "users" are simply the website users and "items" are streamers that they should watch. I ...
0
votes
0answers
9 views

Appropriate naive benchmark for class recall in binary classification for unbalanced dataset

I have an unbalanced dataset with 3969 rows of customer data. The labels are whether or not they subscribed for a loan (yes or no). There are 3618 no cases (91.2%) and 351 yes cases (8.8%). I am more ...
1
vote
1answer
81 views

What am I supposed to see on tensorboard images tab?

I'm training an object detection model with Tensorflow and monitor the training task with tensorboard. I was expecting in the Images tab of tensorboard that displayed images would show a bounding box (...
2
votes
0answers
23 views

Assessing model performance on different sub-segments

I am currently working on a credit risk related project where i built a binary logistic regression model for an imbalanced dataset. According to the regulations i have to prove that the model performs ...
0
votes
1answer
35 views

Machine Learning validation data returns 100% accuracy [closed]

I'm Testing a Machine Learning model with validation data returns that return 100% correct answers, is it overfitting or the model works extremely well, do I need to continue training on more data? I'...
1
vote
1answer
42 views

Time series analysis model evaluation performance metrics integration in time series application

After study in time series analysis, I recognized RMSE and MAPE are the best evaluation metrics for used model in real time series application. But my queries are below as this is my first practice ...
1
vote
0answers
42 views

Match between objective function and evaluation metric

Does the objective function for model fitting and the evaluation metric for model validation need to be identical throughout the hyperparameter search process? For example, can a XGBoost model be ...
0
votes
0answers
8 views

Get latest Item by Date for a Recommender System

I am building a Recommender System where I am giving the User 3 Recommendations depending upon for the Webpage he is on. Let's say My model gives me 3 Recommendations from 2020, 2019, 2015. I would ...
1
vote
1answer
33 views

How do you identify whether your RMSE score is good or not?

Im building a XGBoost regression model to predict the values in the range of -3 to 3. Im using Root Mean Squared Error to evaluate the model. With hyper-parameter tuning and everything the best scores ...
0
votes
1answer
33 views

model selection in clustering

I am working on a mall customer segmentation dataset (5 features, 200 rows) using clustering. This dataset does not have any ground truth labels. I had a few doubts regarding clustering: Can I use ...
1
vote
1answer
34 views

Is it a good practice to evaluate model performance by comparing the metrics of rescaled (inverse transformed) predictions and true target values?

I am now working with a Linear Regression for a time-series regression problem (I am sorry that I cannot say too much about the problem and feature vector due to NDA). I scaled both the input values ...
0
votes
1answer
53 views

How and where to set weights in case of imbalanced cost sensitive learning in machine learning?

I confront with a binary classification machine learning task which is both slightly imbalanced and cost sensitive. I wonder what (and where in the modeling pipeline, say, in sklearn) is the best way ...
1
vote
1answer
20 views

NER evaluation metric

I'm trying to compare two NER tools on an annotated corpus and I'm not sure which is the best metric to use, as I haven't worked with NER models before. To be more specific, I'm interested in one ...
0
votes
1answer
39 views

Why Davies-Bould chose a number ob cluster higher than Silhouette or Calinsky Harabasz?

I am doing use several metrics in order to know what number of clusters is correct in order to do this I selected 3 clustering algorithms and 3 internal evaluation metrics, Silhouette, Calinsky ...
1
vote
0answers
60 views

Fine tuning a automatic speech recognition model with my own dataset

I'm using wav2letter to develop a speech-to-text system. wav2letter has pre-built acoustic and language models which is great, however the audio that I am transcribing from is unique in comparison to ...
0
votes
1answer
37 views

Comparing Dataset - Should I use the same Test dataset?

I am training ML CNN model. I want to compare different images dataset. The dataset all have different characteristics (Translated or not, Rotated or not, etc.). I do not modify the ML model between ...
0
votes
3answers
68 views

Evaluating a Multi-Label Classification model

I currently have a multi-label classification problem, for which I am using keras to build a neural network as follows: ...
2
votes
2answers
120 views

Best common metric for comparing classic time series forecasting methods (ARIMA/Prophet) with ML approach?

I am new to time series forecasting and looking to compare the performance of ARIMA/Prophet with an XGBoost model in predicting future stock market values based on historical stock market data and ...
1
vote
0answers
65 views

Is the micro averaged precision/recall/f1 score for multiclass classification always the same?

I was under the impression from this post that in the micro averaging case for multi-class classification, the precision and recall are the same. This is because the number of false negatives and ...
5
votes
3answers
74 views

Finding out why your model is doing better?

I fitted a logistic regression model on a data set and got an AUC score of .70. I added some additional out-hot encoded categorical features to the model and the AUC improved slightly to .74. How do I ...
3
votes
1answer
133 views

Appropriate objective function and evaluation metric when I DO care about outliers?

I am reading these two pages: xgboost documentation Post on evaluation metrics I have a dataset where I am trying to predict future spend at the user level. A lot of our spend comes from large ...
0
votes
0answers
15 views

spacy train cli compare iterations

So I am running: spacy train da [several] [options] [here] And I am getting a nice overview in the console about the performance of each iteration, in terms of P, ...
0
votes
1answer
20 views

How to derive false positive and false negative from top-k accuracy?

I am working on the following "equality identification" problem and become quite confused on how to reasonably define false positive and false negative in my case. Problem: Suppose I have a ...
0
votes
0answers
58 views

Evaluation of Mixture Density Networks

I have programmed an Mixture Density Network model to a market price. As input I have many numerical and categorical properties. The output of the network is a probability distribution (shape, ...
0
votes
0answers
60 views

How to interpret hard negative mining curves while training a deep object detector?

I am training a single shot detector (SSD) in tensorflow object detection API. After having read the paper and some articles online, I understood that hard negative mining trains the network on 'hard ...
1
vote
0answers
17 views

Orange 3 working with multiple datasets [closed]

I do not know if it is possible to somehow connect different datasets, that are related in some way, in Orange 3. I am working with EEG multiple files (of channel values results) and one main data ...
0
votes
1answer
35 views

Is epsilon error a standard known error or custom created by this paper?

I'm reading this computer vision paper, research paper link, about creating a model to estimate the real age and perceived age of the person in the image (or at least that's what I think it's about). ...
8
votes
2answers
416 views

Do I need validation data if my train and test accuracy/loss is consistent?

I am trying to understand the purpose of a 3rd split in the form of a validation dataset. I am not necessarily talking about cross-validation here. In the scenario below, it would appear that the ...
0
votes
1answer
30 views

What methods are available to evaluate similarity between different clustering algorithms?

I am performing extensive customer segmentation analysis and so far implemented Gaussian Mixture Models, K-Means, and Hierarchical Clustering. For the most part, the algorithms agree on the structure ...
1
vote
0answers
16 views

How do you evaluate a predictive model that is deployed in production that is suppose to prevent certain scenerio?

Let's consider I trained a model that gives the probability if an apple will rot and deployed it. Once it's deployed, how can I measure/evaluate its performance. At first I thought, we can just check ...
1
vote
0answers
14 views

testing statistical significance when comparing regression models

I have 3 models, a random forest, a XGBoost and a baseline regression model. I've performed a 5-fold cross validation on all models and use MAE as the scoring metric. So this means I basically have 3 ...
1
vote
1answer
35 views

Loading a Keyword and Evaluating the Information

I am an Ex Service Veteran and need assistance with a Small Program to use with my Rats of Tobruk Project, for the purpose of evaluating Archives Information, which I normally do by Manual Means, but ...
0
votes
0answers
14 views

Public benchmark datasets posted with expected/record scores for sanity check?

When I use a new modelling tool or approach, I like to do a quick sanity check on a public dataset to make sure I'm getting good (but not "so good it looks fishy") scores. There are several clean, ...
0
votes
0answers
10 views

Model Evaluation Documentation

I have been training a simple model against different machine learning algorithms : logistic, random forest, decision tree, xgboost and adaboost. I have gathered metrics about accuracy, roc auc, ...
1
vote
1answer
201 views

What is the concept of Normalized Mutual Information in the evaluation of Clustering?

I know what mutual information basically is but not quite sure about why and how it is used in the context of evaluation of clustering mechanisms ? Can someone please explain the intuition behind it ? ...
1
vote
1answer
280 views

Evaluate Keras model with scikit-learn metrics

How does Keras calculate accuracy, precision, recall, and AUC? I've created a model for categorical classification (i.e., multiple classes) by using ...
0
votes
1answer
100 views

Confusion matrix of UNET image sgemenation model

I have used Unet model for image segmentation. I have used RGB images and corresponding image masks and at output i got corresponding region of interest. Now i want to find confusion matrix of this ...
0
votes
0answers
26 views

Predict best score on unlabelled test set

Data I have one dataset with $1500$ data points, each with $\sim 23 000$ features (gene expression data, if that matters). However, I've split this dataset into a labelled training set of size 1000, ...

1
2 3 4 5