Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Get latest Item by Date for a Recommender System

I am building a Recommender System where I am giving the User 3 Recommendations depending upon for the Webpage he is on. Let's say My model gives me 3 Recommendations from 2020, 2019, 2015. I would ...
0
votes
0answers
13 views

Evaluation test on intent classification: precision, recall, f1-score [closed]

I’m running an evaluation test on intent classification with supervised embeddings using the following code: ...
-2
votes
0answers
25 views

Are these precision recall curves possible where single curve intersects each other? [closed]

Someone just showed me this precision-recall curve for decision tree classifiers after feeding scores and targets to scikit learns precision-recall curve module. Is this possible that the single curve ...
1
vote
1answer
27 views

How do you identify whether your RMSE score is good or not?

Im building a XGBoost regression model to predict the values in the range of -3 to 3. Im using Root Mean Squared Error to evaluate the model. With hyper-parameter tuning and everything the best scores ...
0
votes
1answer
29 views

model selection in clustering

I am working on a mall customer segmentation dataset (5 features, 200 rows) using clustering. This dataset does not have any ground truth labels. I had a few doubts regarding clustering: Can I use ...
1
vote
1answer
29 views

Is it a good practice to evaluate model performance by comparing the metrics of rescaled (inverse transformed) predictions and true target values?

I am now working with a Linear Regression for a time-series regression problem (I am sorry that I cannot say too much about the problem and feature vector due to NDA). I scaled both the input values ...
0
votes
1answer
21 views

How and where to set weights in case of imbalanced cost sensitive learning in machine learning?

I confront with a binary classification machine learning task which is both slightly imbalanced and cost sensitive. I wonder what (and where in the modeling pipeline, say, in sklearn) is the best way ...
1
vote
1answer
17 views

NER evaluation metric

I'm trying to compare two NER tools on an annotated corpus and I'm not sure which is the best metric to use, as I haven't worked with NER models before. To be more specific, I'm interested in one ...
0
votes
1answer
35 views

Why Davies-Bould chose a number ob cluster higher than Silhouette or Calinsky Harabasz?

I am doing use several metrics in order to know what number of clusters is correct in order to do this I selected 3 clustering algorithms and 3 internal evaluation metrics, Silhouette, Calinsky ...
1
vote
0answers
31 views

Fine tuning a automatic speech recognition model with my own dataset

I'm using wav2letter to develop a speech-to-text system. wav2letter has pre-built acoustic and language models which is great, however the audio that I am transcribing from is unique in comparison to ...
0
votes
1answer
28 views

Comparing Dataset - Should I use the same Test dataset?

I am training ML CNN model. I want to compare different images dataset. The dataset all have different characteristics (Translated or not, Rotated or not, etc.). I do not modify the ML model between ...
0
votes
2answers
26 views

Evaluating a Multi-Label Classification model

I currently have a multi-label classification problem, for which I am using keras to build a neural network as follows: ...
2
votes
2answers
35 views

Best common metric for comparing classic time series forecasting methods (ARIMA/Prophet) with ML approach?

I am new to time series forecasting and looking to compare the performance of ARIMA/Prophet with an XGBoost model in predicting future stock market values based on historical stock market data and ...
1
vote
0answers
28 views

Is the micro averaged precision/recall/f1 score for multiclass classification always the same?

I was under the impression from this post that in the micro averaging case for multi-class classification, the precision and recall are the same. This is because the number of false negatives and ...
4
votes
1answer
49 views

Finding out why your model is doing better?

I fitted a logistic regression model on a data set and got an AUC score of .70. I added some additional out-hot encoded categorical features to the model and the AUC improved slightly to .74. How do I ...
0
votes
0answers
19 views

Backtesting - Multiple Train-Test Splits

I am looking to use an auto ML platform for retail forecasting. Will use 3.5 years of sales data. Our business has changed significantly. Higher margins, fewer incentives and competition has ...
3
votes
1answer
60 views

Appropriate objective function and evaluation metric when I DO care about outliers?

I am reading these two pages: xgboost documentation Post on evaluation metrics I have a dataset where I am trying to predict future spend at the user level. A lot of our spend comes from large ...
0
votes
0answers
12 views

spacy train cli compare iterations

So I am running: spacy train da [several] [options] [here] And I am getting a nice overview in the console about the performance of each iteration, in terms of P, ...
0
votes
1answer
20 views

How to derive false positive and false negative from top-k accuracy?

I am working on the following "equality identification" problem and become quite confused on how to reasonably define false positive and false negative in my case. Problem: Suppose I have a ...
0
votes
0answers
51 views

Evaluation of Mixture Density Networks

I have programmed an Mixture Density Network model to a market price. As input I have many numerical and categorical properties. The output of the network is a probability distribution (shape, ...
0
votes
0answers
24 views

How to interpret hard negative mining curves while training a deep object detector?

I am training a single shot detector (SSD) in tensorflow object detection API. After having read the paper and some articles online, I understood that hard negative mining trains the network on 'hard ...
1
vote
0answers
8 views

Orange 3 working with multiple datasets [closed]

I do not know if it is possible to somehow connect different datasets, that are related in some way, in Orange 3. I am working with EEG multiple files (of channel values results) and one main data ...
0
votes
1answer
31 views

Is epsilon error a standard known error or custom created by this paper?

I'm reading this computer vision paper, research paper link, about creating a model to estimate the real age and perceived age of the person in the image (or at least that's what I think it's about). ...
8
votes
2answers
396 views

Do I need validation data if my train and test accuracy/loss is consistent?

I am trying to understand the purpose of a 3rd split in the form of a validation dataset. I am not necessarily talking about cross-validation here. In the scenario below, it would appear that the ...
0
votes
1answer
17 views

What methods are available to evaluate similarity between different clustering algorithms?

I am performing extensive customer segmentation analysis and so far implemented Gaussian Mixture Models, K-Means, and Hierarchical Clustering. For the most part, the algorithms agree on the structure ...
1
vote
0answers
13 views

How do you evaluate a predictive model that is deployed in production that is suppose to prevent certain scenerio?

Let's consider I trained a model that gives the probability if an apple will rot and deployed it. Once it's deployed, how can I measure/evaluate its performance. At first I thought, we can just check ...
1
vote
0answers
14 views

testing statistical significance when comparing regression models

I have 3 models, a random forest, a XGBoost and a baseline regression model. I've performed a 5-fold cross validation on all models and use MAE as the scoring metric. So this means I basically have 3 ...
1
vote
1answer
35 views

Loading a Keyword and Evaluating the Information

I am an Ex Service Veteran and need assistance with a Small Program to use with my Rats of Tobruk Project, for the purpose of evaluating Archives Information, which I normally do by Manual Means, but ...
0
votes
0answers
14 views

Public benchmark datasets posted with expected/record scores for sanity check?

When I use a new modelling tool or approach, I like to do a quick sanity check on a public dataset to make sure I'm getting good (but not "so good it looks fishy") scores. There are several clean, ...
0
votes
0answers
9 views

Model Evaluation Documentation

I have been training a simple model against different machine learning algorithms : logistic, random forest, decision tree, xgboost and adaboost. I have gathered metrics about accuracy, roc auc, ...
1
vote
1answer
28 views

What is the concept of Normalized Mutual Information in the evaluation of Clustering?

I know what mutual information basically is but not quite sure about why and how it is used in the context of evaluation of clustering mechanisms ? Can someone please explain the intuition behind it ? ...
1
vote
1answer
65 views

Evaluate Keras model with scikit-learn metrics

How does Keras calculate accuracy, precision, recall, and AUC? I've created a model for categorical classification (i.e., multiple classes) by using ...
0
votes
0answers
40 views

Confusion matrix of UNET image sgemenation model

I have used Unet model for image segmentation. I have used RGB images and corresponding image masks and at output i got corresponding region of interest. Now i want to find confusion matrix of this ...
0
votes
0answers
22 views

Predict best score on unlabelled test set

Data I have one dataset with $1500$ data points, each with $\sim 23 000$ features (gene expression data, if that matters). However, I've split this dataset into a labelled training set of size 1000, ...
0
votes
0answers
16 views

Benchmark queries for Benchmark dataset with ranked list of documents

I aim to evaluate the ranking of an information-retrieval system. For a benchmark dataset like TREC, I have followed the qrels file which has list of documents for a particular topic (query) with ...
0
votes
1answer
21 views

How to compare supervised learning algorithm and it's technique ensemble learning algorithm?

I have to compare Support Vector Machine and Random Forest algorithm , but i'm confused how it can be compared, like support vector machine is supervised learning algorithm and random forest is ...
1
vote
0answers
11 views

Calculating precision with concordance matrix

I'm calculating some external validity indices for clustering like the Folkes-Mallows, Adjusted Rand Index, Jaccard, Rand, etc.. I'm using the R package clusterCrit (link to pdf) to determinate this ...
0
votes
0answers
6 views

How to evaluate personalized model with indifferent labels?

I am currently working on a project for measuring moods based on the data collected from wearable sensors. When I wanted to develop a personalized model based on different participants, I noticed that ...
-1
votes
1answer
57 views

What is evaluation metric for two sets? [closed]

I've two sets one is ground truth and other is output of my machine learning models. Assume my groundtruth set is A={1,2,3,4,5} and output of machine learning model is B={3,4,5,6,7,8}. One way I can ...
0
votes
0answers
36 views

Multiple choice gap-fill question (with distractors) dataset for evaluating NLP algorithms

I am looking for a standard gap-filling multiple-choice exercise (with distractors) dataset that can be used to evaluate the NLP gap-filling ML algorithms. I expect the dataset to contain questions ...
0
votes
0answers
13 views

Approximate evaluation for deep learning architecture

When train on big dataset for deep learning architecture, like imagenet, it takes long time to judge whether our new neural network architecture is good, say for image classification. Is there a way ...
1
vote
1answer
37 views

Offline evaluation of recommender systems

Let's say I want to compare whether one recommender system (A) is better than the other (B). One approach is to let people rate recommendations returned by both systems. However, there situations ...
1
vote
1answer
19 views

Measuring impact of missing feature on model performance

Is there any general approach to approximate how a missing feature will impact the prediction performance of a regression model? For example, if I train a model using 10 features, but want to make ...
1
vote
3answers
63 views

Precision and Recall Confusion matrix

I was wondering, is it a proper method to convey information via separate Recall and Precision Confusion matrix? I recently came across a paper which reported the following scores. I am puzzled and ...
0
votes
2answers
68 views

How I can compute Matthews Correlation Coefficient?

I want to calculate MCC, I have two CSV files: result.csv and true.csv result.csv format: ...
3
votes
2answers
37 views

Is it correct to define the F-measure as the harmonic mean of specificity and sensitivity in such a way?

It is common to define the F-measure as a function of precision and recall, as mentioned in [1]: $F_{\beta}=\frac{(1+\beta^2)PR}{\beta^2 P+R}$ However I came across some other cases, another ...
1
vote
1answer
29 views

Is there an established methodology for human expert evaluation of machine learning predictions?

Assume a continuous dependent variable $Y$. Then, $\hat{Y}$ is the predictions made by a machine learning-based model using a set of independent variables $X$. In this case, however, it is known ...
1
vote
1answer
29 views

How to select optimal threshold which separate different classes?

I have trained a network to find the similarity between two images. The test dataset contain equal number of similar and dissimilar samples. Each class has approx. 13822 samples. I tried different ...
0
votes
0answers
20 views

Reinforcement Learning : Why acting greedily with the optimal value function gives you the optimal policy?

The course of David Silver about Reinforcement Learning explains how you get the optimal policy from the optimal value function. It seems to be very simple, you just have to act greedily, by ...
1
vote
1answer
31 views

what metrics to evaluate rank order results?

I have searched on stackexchange and found a couple of topics like this and this but they are not quite relevant to my problem (or at least I don't know how to make them relevant to my problem). ...

1
2 3 4 5