Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Backtesting Windows

I am looking to use an auto ML platform for retail forecasting. Will use 3.5 years of sales data. Our business has changed significantly. Higher margins, fewer incentives and competition has ...
0
votes
1answer
20 views

Appropriate objective function and evaluation metric when I DO care about outliers?

I am reading these two pages: xgboost documentation Post on evaluation metrics I have a dataset where I am trying to predict future spend at the user level. A lot of our spend comes from large ...
0
votes
0answers
7 views

spacy train cli compare iterations

So I am running: spacy train da [several] [options] [here] And I am getting a nice overview in the console about the performance of each iteration, in terms of P, ...
0
votes
1answer
20 views

How to derive false positive and false negative from top-k accuracy?

I am working on the following "equality identification" problem and become quite confused on how to reasonably define false positive and false negative in my case. Problem: Suppose I have a ...
0
votes
0answers
45 views

Evaluation of Mixture Density Networks

I have programmed an Mixture Density Network model to a market price. As input I have many numerical and categorical properties. The output of the network is a probability distribution (shape, ...
0
votes
0answers
15 views

How to interpret hard negative mining curves while training a deep object detector?

I am training a single shot detector (SSD) in tensorflow object detection API. After having read the paper and some articles online, I understood that hard negative mining trains the network on 'hard ...
1
vote
0answers
8 views

Orange 3 working with multiple datasets [closed]

I do not know if it is possible to somehow connect different datasets, that are related in some way, in Orange 3. I am working with EEG multiple files (of channel values results) and one main data ...
0
votes
1answer
28 views

Is epsilon error a standard known error or custom created by this paper?

I'm reading this computer vision paper, research paper link, about creating a model to estimate the real age and perceived age of the person in the image (or at least that's what I think it's about). ...
7
votes
2answers
382 views

Do I need validation data if my train and test accuracy/loss is consistent?

I am trying to understand the purpose of a 3rd split in the form of a validation dataset. I am not necessarily talking about cross-validation here. In the scenario below, it would appear that the ...
0
votes
1answer
13 views

What methods are available to evaluate similarity between different clustering algorithms?

I am performing extensive customer segmentation analysis and so far implemented Gaussian Mixture Models, K-Means, and Hierarchical Clustering. For the most part, the algorithms agree on the structure ...
1
vote
0answers
12 views

How do you evaluate a predictive model that is deployed in production that is suppose to prevent certain scenerio?

Let's consider I trained a model that gives the probability if an apple will rot and deployed it. Once it's deployed, how can I measure/evaluate its performance. At first I thought, we can just check ...
1
vote
0answers
13 views

testing statistical significance when comparing regression models

I have 3 models, a random forest, a XGBoost and a baseline regression model. I've performed a 5-fold cross validation on all models and use MAE as the scoring metric. So this means I basically have 3 ...
1
vote
1answer
35 views

Loading a Keyword and Evaluating the Information

I am an Ex Service Veteran and need assistance with a Small Program to use with my Rats of Tobruk Project, for the purpose of evaluating Archives Information, which I normally do by Manual Means, but ...
0
votes
0answers
14 views

Public benchmark datasets posted with expected/record scores for sanity check?

When I use a new modelling tool or approach, I like to do a quick sanity check on a public dataset to make sure I'm getting good (but not "so good it looks fishy") scores. There are several clean, ...
0
votes
0answers
8 views

Model Evaluation Documentation

I have been training a simple model against different machine learning algorithms : logistic, random forest, decision tree, xgboost and adaboost. I have gathered metrics about accuracy, roc auc, ...
1
vote
1answer
14 views

What is the concept of Normalized Mutual Information in the evaluation of Clustering?

I know what mutual information basically is but not quite sure about why and how it is used in the context of evaluation of clustering mechanisms ? Can someone please explain the intuition behind it ? ...
1
vote
1answer
40 views

Evaluate Keras model with scikit-learn metrics

How does Keras calculate accuracy, precision, recall, and AUC? I've created a model for categorical classification (i.e., multiple classes) by using ...
0
votes
0answers
17 views

Confusion matrix of UNET image sgemenation model

I have used Unet model for image segmentation. I have used RGB images and corresponding image masks and at output i got corresponding region of interest. Now i want to find confusion matrix of this ...
0
votes
0answers
19 views

Predict best score on unlabelled test set

Data I have one dataset with $1500$ data points, each with $\sim 23 000$ features (gene expression data, if that matters). However, I've split this dataset into a labelled training set of size 1000, ...
0
votes
0answers
15 views

Benchmark queries for Benchmark dataset with ranked list of documents

I aim to evaluate the ranking of an information-retrieval system. For a benchmark dataset like TREC, I have followed the qrels file which has list of documents for a particular topic (query) with ...
0
votes
1answer
20 views

How to compare supervised learning algorithm and it's technique ensemble learning algorithm?

I have to compare Support Vector Machine and Random Forest algorithm , but i'm confused how it can be compared, like support vector machine is supervised learning algorithm and random forest is ...
1
vote
0answers
7 views

Calculating precision with concordance matrix

I'm calculating some external validity indices for clustering like the Folkes-Mallows, Adjusted Rand Index, Jaccard, Rand, etc.. I'm using the R package clusterCrit (link to pdf) to determinate this ...
0
votes
0answers
5 views

How to evaluate personalized model with indifferent labels?

I am currently working on a project for measuring moods based on the data collected from wearable sensors. When I wanted to develop a personalized model based on different participants, I noticed that ...
-1
votes
1answer
52 views

What is evaluation metric for two sets? [closed]

I've two sets one is ground truth and other is output of my machine learning models. Assume my groundtruth set is A={1,2,3,4,5} and output of machine learning model is B={3,4,5,6,7,8}. One way I can ...
0
votes
0answers
35 views

Multiple choice gap-fill question (with distractors) dataset for evaluating NLP algorithms

I am looking for a standard gap-filling multiple-choice exercise (with distractors) dataset that can be used to evaluate the NLP gap-filling ML algorithms. I expect the dataset to contain questions ...
0
votes
0answers
12 views

Approximate evaluation for deep learning architecture

When train on big dataset for deep learning architecture, like imagenet, it takes long time to judge whether our new neural network architecture is good, say for image classification. Is there a way ...
1
vote
1answer
29 views

Offline evaluation of recommender systems

Let's say I want to compare whether one recommender system (A) is better than the other (B). One approach is to let people rate recommendations returned by both systems. However, there situations ...
1
vote
1answer
17 views

Measuring impact of missing feature on model performance

Is there any general approach to approximate how a missing feature will impact the prediction performance of a regression model? For example, if I train a model using 10 features, but want to make ...
1
vote
3answers
58 views

Precision and Recall Confusion matrix

I was wondering, is it a proper method to convey information via separate Recall and Precision Confusion matrix? I recently came across a paper which reported the following scores. I am puzzled and ...
0
votes
2answers
43 views

How I can compute Matthews Correlation Coefficient?

I want to calculate MCC, I have two CSV files: result.csv and true.csv result.csv format: ...
3
votes
2answers
36 views

Is it correct to define the F-measure as the harmonic mean of specificity and sensitivity in such a way?

It is common to define the F-measure as a function of precision and recall, as mentioned in [1]: $F_{\beta}=\frac{(1+\beta^2)PR}{\beta^2 P+R}$ However I came across some other cases, another ...
1
vote
1answer
26 views

Is there an established methodology for human expert evaluation of machine learning predictions?

Assume a continuous dependent variable $Y$. Then, $\hat{Y}$ is the predictions made by a machine learning-based model using a set of independent variables $X$. In this case, however, it is known ...
1
vote
1answer
27 views

How to select optimal threshold which separate different classes?

I have trained a network to find the similarity between two images. The test dataset contain equal number of similar and dissimilar samples. Each class has approx. 13822 samples. I tried different ...
0
votes
0answers
17 views

Reinforcement Learning : Why acting greedily with the optimal value function gives you the optimal policy?

The course of David Silver about Reinforcement Learning explains how you get the optimal policy from the optimal value function. It seems to be very simple, you just have to act greedily, by ...
1
vote
1answer
27 views

what metrics to evaluate rank order results?

I have searched on stackexchange and found a couple of topics like this and this but they are not quite relevant to my problem (or at least I don't know how to make them relevant to my problem). ...
0
votes
0answers
10 views

How to select checkpoint for model evaluation?

I have trained a deep convolutional neural network for image similarity classification. The network returns whether the images are the same or different. I trained the network for 20 epochs and save ...
0
votes
2answers
34 views

Manual way to draw accuracy/loss graphs

During the training process of the convolutional neural network, the network outputs the training/validation accuracy/loss after each epoch as shown below: ...
2
votes
1answer
136 views

How to correctly calculate average F1 score, precision and recall of a Named Entity Recognition system?

My Named Entity Recognition (NER) pipeline built with Apache uimaFIT and DKPro recognizes named entities (called datatypes for now) in texts (e.g. persons, locations, organizations and many more). I ...
2
votes
0answers
51 views

Confused about the MSE ERROR

I am created a random forest regressor and calculate my own error. I want also to calculate MAE, MSE and RMSE to compare my results to similar usecases. But the results of the MAE, MSE, RMSE are ...
1
vote
1answer
75 views

How to keep the test data from leaking into the training process of a machine learning algorithm?

I read in many different sources that I need to split my data into a training set and a test set. Then I have to make sure that the algorithm is trained only on the training data, and do my best to ...
1
vote
1answer
44 views

How to estimate the accuracy on a large dataset?

Given that I have a deep learning model(handover from former colleague). For some reason, the train/dev set was missing. In my situation, I want to classify my dataset into 100 categories. The ...
1
vote
1answer
278 views

Need of Weighted Mean Squared Error

We have MSE and RMSE as evaluation metrics for regression problems. I have for some problems people use Weighted Mean Squared Error (WMSE) as the evaluation metrix. Below is the WMSE formula: Can ...
2
votes
0answers
23 views

FFR and FAR calculating for multiclasss biometric face recognition system

I am implementing a face recognition system using facenet and svc Ml algorithm i have like 20 classes or more and I'm getting 98% accuracy im trying to calculate the FAR and FRR and the EER I'm ...
2
votes
1answer
22 views

evaluation metrics for multiple values per session

I have an application that executes my foo() function several times for each user session. There are 2 alternate algorithms that i can implement as "foo" function and my goal is to evaluate them based ...
1
vote
1answer
387 views

XGBoost Feature Importance, Permutation Importance, and Model Evaluation Criteria

I have built an XGBoost classification model in Python on an imbalanced dataset (~1 million positive values and ~12 million negative values), where the features are binary user interaction with web ...
1
vote
1answer
136 views

How to Maximize recall for Minority class?

I have a dataset with 4.7k records and 60 features. 1558 records of indication label 1 and 3554 records indicating label 0. Am ...
3
votes
2answers
63 views

How best to show the best model over multiple labels?

I have 4 models I trained and I want to display their prediction success over 45 different labels I tested them on. I get a very messy plot when I naively try to place them one on top of the other. ...
2
votes
1answer
127 views

Calculating Rank Ordering Error Metric for implicit recommendation

I'm reading Collaborative Filtering for Implicit Feedback Datasets. On page 6 they detail their evaluation strategy, which they define as mean Expected Percentile Ranking with the following formula: $...
0
votes
0answers
45 views

taking np.argmax while evaluating the model

I'm studying a code for a task of Music Genre Classification and I'm stuck at understanding a few line of codes that come after the model has been built. Basically it concerns the valuation of your ...
4
votes
1answer
42 views

How is “relevance” defined in information retrieval outside the context of systems with user feedback?

I've seen information retrieval systems that return some results from a query, and then the user rates these results as either "relevant" or "not relevant". What can you do if you do not have user ...