Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

Filter by
Sorted by
Tagged with
1
vote
1answer
406 views

Is recall more important than precision for mass mailings?

Say for example, I built a classification model for a mailing campaign that will be applied to 1M records. The positive class for the model would be customers and the negative records would be non-...
0
votes
1answer
858 views

Python - Calculate Cost profitability and benefit of the model

I've this code in Python in order to calculate the precision of my model and to print confusion matrix using Decision Trees Classifier: ...
1
vote
1answer
261 views

What is the advantage of using Dunn index over other metrics for evaluating clustering algorithm? [closed]

There are many metrics to evaluate clustering algorithm like Calinski-Harabaz Index, Dunn index, Rand index, etc. Are there any advantage of using Dunn index over other metrics for evaluating ...
1
vote
0answers
90 views

Performance Metric for topic extraction when there is no ground truth

I am extracting topics from text using a predefined ontology containing 2690 concepts, wordnet(to expand concept terms with their synsets, and other morphological forms of the same word) and lucene to ...
2
votes
1answer
21 views

Find threshold in rate to determine reason for lost customer

I'm not sure if anybody will be able to help with this, or even if I'll be able to explain it well but I am stuck so here goes.... I have a set of customers, some are lost, some are still active. We ...
4
votes
3answers
370 views

Evaluating new features

How should I evaluate whether new features are effective or not? Should I build a new model with the new features then compare with the old one with the same hyper parameter?
0
votes
1answer
905 views

How to measure F1 score and NMI for clustering task?

I see the authors of this paper are measuring F1 and NMI scores to measure the clustering quality. However, I don't understand the algorithm of how they actually measure it. See the Evaluation Section....
2
votes
0answers
21 views

Do I need to use Bayes to combine a sample's class probability with the performance of the overall model?

For a classification algorithm that gives the predicted probabilities for each class (ie random forest in sklearn), by default the classes are separated with a score of 0.5. After evaluating the ...
0
votes
1answer
50 views
2
votes
1answer
2k views

Splitting hold-out sample and training sample only once?

I have a question related to evaluating out-of-sample predictions. For my research I want to tune two parameters related to Support Vector Machines, and use these optimized parameters to predict the ...
0
votes
1answer
200 views

Clustering documents - how to evaluate results?

I'm using DBSCAN clustering on a set of documents. The documents' content was converted to TF-IDF matrix, and I'd like to find consistent ways to evaluate the clusters when no added information is ...
1
vote
1answer
215 views

How to evaluate sequence to sequence models?

I wonder how to evaluate variable long sequence-to-sequence predictions? Let us say I have the following $Y$ and $\hat{Y}$ $Y = [["1", "2", "2"], ["3", "2", "2"], ["1", "3", "2", "2"]]$ $\hat{Y} = [[...
4
votes
1answer
150 views

How can RL agents be monitored?

My question is about how to monitor RL agents in production. To make the question easier to discuss, here is a use case. Please don't focus on difficulties in implementing such an agent, but rather on ...
1
vote
1answer
183 views

How to evaluate multi label image retrieval model

I'm using a deep hashing model to search most similar images in a database (most similar to the image given as a query). I'm doing this on the coco dataset which has multiple labels per image. I'd ...
3
votes
1answer
2k views

Irregular Precision-Recall Curve

I'd expect that for a precision-recall curve, precision decreases while recall increases monotonically. I have a plot that is not smooth and looks funny. I used scikit learn the values for plotting ...
-1
votes
1answer
251 views

What are the drawbacks of V-measure clustering evaluation method?

What are the drawbacks of V-measure clustering evaluation method? For evaluating what clustering algorithms, is the V-measure evaluation method suitable?
0
votes
1answer
67 views

How to get an intuitive value for regression module evaluation?

For regression module evaluation, I think only the MAE (Mean absolute error) value is not objective or practical. Consider following situations: A MAE=1 while ...
0
votes
2answers
202 views

recommender system: how to compare different scores when calculated individually?

I am building a small recommender system which aims at recommending ~10 products to customers. Instead of using a multi-label classification model, I have opted to build a separate scoring model for ...
0
votes
1answer
228 views

Metrics show badly performing model for multiclass

So I have a model that I am training on a multiclass (30-40 classes) imbalanced data set (smallest class 4000 samples, largest 14 million). The data consists of strings and I extract unigram and ...
0
votes
2answers
678 views

In sklearn's classification report, is f1 the best accuracy measure?

In the classification_report provided by sklearn, which score should I look at to make the best determination of the accuracy of my model? ...
13
votes
1answer
8k views

How many features to sample using Random Forests

The Wikipedia page which quotes "The Elements of Statistical Learning" says: Typically, for a classification problem with $p$ features, $\lfloor \sqrt{p}\rfloor$ features are used in each split. ...
1
vote
0answers
40 views

What does NIST information weights refer to?

NIST is a metric used to measure the goodness of translation. In the paper, Doddington (2002) introduce the notion of "Information weights" Information weights were computed using N-gram counts ...
2
votes
2answers
2k views

Why exactly using a test set for model evaluation is a bad idea?

I don't understand why using the test set for model evaluation is a bad idea. I completely understand why you should not use your test set to train your model (because in that case, you would be ...
1
vote
0answers
2k views

Can tuning individual precision and recall classification thresholds improve deep learning models?

I learned that Keras doesn't have a built-in way to set a threshold for precision and accuracy when building a classifier. Courtesy of a solution here, I wanted to see what would happen when I fit a ...
3
votes
1answer
100 views

What is the efficiency difference between different cost functions in case of neural networks?

I'm studying the theory behind neural nets and I wonder if there is any actual difference between using different lost/cost functions? Let's say I could use either MAE or MSE for backpropagating loss;...
1
vote
1answer
2k views

python xgboost DMatrix - get feature values or convert to np.array

I'm trying to create a custom evaluation metric (feval) function for xgboost.cv. It should process some of the training features, however I can't find a way to extract features from ...
0
votes
1answer
108 views

Classifier runtime evaluation

I am looking for an example or tutorial of a system predicting numeric values by the use of various classifiers like SVM, Decision trees, ANN or KNN which optimises its choice of algorithm and ...
3
votes
1answer
843 views

Evaluating Logistic Regression Model in Tensorflow

Following this tutorial, I have a doubt about the evaluation part in: ...
2
votes
3answers
7k views

Is Gini coefficient a good metric for measuring predictive model performance on highly imbalanced data

I am evaluating a Credit Risk model that predicts the estimated likelihood of customers defaulting on their mortgage accounts. The model is a Logistic Regression estimator and was built by another ...
2
votes
1answer
2k views

How to compare LDA and TF-IDF?

I am doing text mining to extract topics from documents. I started with Latent Dirichlet Allocation (LDA), which worked great, but then I came across TF-IDF with K-Means clustering, which worked ...
1
vote
1answer
538 views

How to compare performance of Cosine Similarity and Manhatten Distance?

I'm doing clustering of documents by applying k-Means on the word-vectors. To measure the cluster quality, I calculate David Bouldin Index for different k's. I tried two different distance measures, ...
3
votes
0answers
1k views

In XGBoost, how to change eval function and keeping same objective?

I want to keep objective as "reg:linear" and eval_metric as customised rmse as follows. ...
-1
votes
1answer
1k views

How much should I pay attention to the f1 score on this case?

I trained a model with results as below. It is a stacking model with base learners of random forest and gradient boosting. The mega model is a GLM. The dataset is imbalanced in the target class as ...
127
votes
6answers
132k views

Micro Average vs Macro average Performance in a Multiclass classification setting

I am trying out a multiclass classification setting with 3 classes. The class distribution is skewed with most of the data falling in 1 of the 3 classes. (class labels being 1,2,3, with 67.28% of the ...
2
votes
1answer
265 views

Comparing Non-deterministic Binary Classifiers

I have two classifiers which I am implementing, and they are both non-deterministic in the sense that they can each give different results (FPR and TPR) when you run them multiple times. I would like ...
1
vote
1answer
105 views

How to test People similarity measure?

I am doing a project on finding famous people who are similar to each other. For this, I am extracting a bunch of features and applying a distance function on them to evaluate who is closer to whom. ...
1
vote
1answer
56 views

How can I fix this “convex” problem ? Is it just a matter of overfitting?

I get some metrics on validation data while training a model , and in my case the they are : (0.25, 0.31, 0.46, 0.57, 0.65, 0.75, 0.77, 0.78, 0.84, 0.84, 0.85, 0.84, 0.84, 0.84, 0.82, 0.8, 0.8, 0....
8
votes
2answers
7k views

Train/Test Split after perform SMOTE

I am dealing with a highly unbalanced data, so I used the SMOTE algorithm to resample the dataset. After SMOTE resampling, I splitted the resampled dataset to training/testing sets, using the ...
7
votes
3answers
3k views

How do you evaluate ML model already deployed in production?

so to be more clear lets consider the problem of loan default prediction. Let's say I have trained and tested off-line multiple classifiers and ensembled them. Then I gave this model to production. ...
3
votes
1answer
2k views

roc_auc score GridSearch

I am experimenting with xgboost. I ran GridSearchCV with score='roc_auc' on xgboost. The best classificator scored ~0.935 (this is what I read from GS output). But now when I run best classificator ...
0
votes
4answers
2k views

Plotting Precision Recall Curve

I was wondering mathematically how Precision Recall curve is plotted. How is this curve useful?
1
vote
1answer
795 views

How to get the inertia at the begining when using sklearn.cluster.KMeans and MiniBatchKMeans

When I cluster a lot of data, it is hard to run KMeans and wait it stop until centers has not change, so I have to stop KMeans when it reach maximum number of iterations. Here come problem: how can I ...
2
votes
0answers
821 views

How to represent ROC curve when using Cross-Validation

I am performing k-Fold Cross Validation using a Logistic Regression classifier on a dataset and computing the ROC curve and the AUC for each fold. My desired output is one ROC curve with a ...
1
vote
0answers
34 views

Can an algorithm tested only on artificial data be accepted in a high rank conference? [closed]

I devised a classification algorithm which is useful for specific complicated distribution of classes of data. The method works good on the artificial data which cannot be solved by typical algorithms ...
2
votes
1answer
846 views

how to evaluate top n recommendation system with movie lens dataset?

Based on my research a recommendation system are a subclass of information filtering system that seek to predict the "rating" or "preference" that a user would give to an item. And I'm currently ...
1
vote
0answers
242 views

Using tensorflow to test a variable amount of correct labels

I'm using a neural network to analyze item choices made by players in a computer game. In the game players can choose between 0 and 7 items. Right now I'm struggling with how I can evaluate my data. ...
4
votes
4answers
295 views

How can conclusions be drawn from recommendation systems evaluation?

From my research, a recommendation system are a subclass of information filtering system that seek to predict the "rating" or "preference" that a user would give to an item. And basically exists many ...
0
votes
2answers
66 views

What do this Classification evaluation results mean to you? Do they are suspicious or not?

I have collected dataset with two class labels and used the SVM Method to classify the dataset, and this is the results. Does this appear suspicious or not? scikit-learn classifiers with SVM SVC ...
11
votes
1answer
21k views

How to define a custom performance metric in Keras?

I tried to define a custom metric fuction (F1-Score) in Keras (Tensorflow backend) according to the following: ...
10
votes
3answers
8k views

Neural Networks - Loss and Accuracy correlation

I'm a bit confused by the coexistence of Loss and Accuracy metrics in Neural Networks. Both are supposed to render the "exactness" of the comparison of $y$ and $\hat{y}$, aren't they? So isn't the ...