Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

Filter by
Sorted by
Tagged with
127
votes
6answers
132k views

Micro Average vs Macro average Performance in a Multiclass classification setting

I am trying out a multiclass classification setting with 3 classes. The class distribution is skewed with most of the data falling in 1 of the 3 classes. (class labels being 1,2,3, with 67.28% of the ...
0
votes
1answer
114 views

How to evaluate clusters base on a label?

I have a data set that has an attribute(A) with 300 different nominal values. Attribute A has a lot of noise. I decide to cluster my data based on other attributes that related to A. I hope to reach ...
13
votes
1answer
8k views

How many features to sample using Random Forests

The Wikipedia page which quotes "The Elements of Statistical Learning" says: Typically, for a classification problem with $p$ features, $\lfloor \sqrt{p}\rfloor$ features are used in each split. ...
11
votes
1answer
21k views

How to define a custom performance metric in Keras?

I tried to define a custom metric fuction (F1-Score) in Keras (Tensorflow backend) according to the following: ...
4
votes
3answers
370 views

Evaluating new features

How should I evaluate whether new features are effective or not? Should I build a new model with the new features then compare with the old one with the same hyper parameter?
3
votes
3answers
942 views

Evaluation methods for multi-class classification

I am looking for single-number evaluation method that can be used in multi-class classification tasks that take into account imbalanced data-sets. For instance, ...
5
votes
1answer
54 views

Evaluating the performance of a machine learned recommendation system

I have a set of resumes $R=\{{r_1,...,r_n\}}$, which I've transformed to a vector space using TF-IDF. Each resume has a label, which is the name of their current employer. Each of these labels comes ...
3
votes
2answers
350 views

In k-fold-cross-validation, why do we compute the mean of the metric of each fold

In k-fold-cross-validation, the "correct" scheme seem to compute the metric (say the accuracy) for each fold, and then return the mean as the final metric. Source : https://scikit-learn.org/stable/...
1
vote
1answer
98 views

How to evaluate clusters base on an attribute of the dataset? [duplicate]

I have a data set of persons with attribute job that have 300 different nominal value. attribute job have a lot of noise. I decide to cluster my data base on other attribute (other feature of person) ...
0
votes
2answers
38 views

Validate via predict() or via fit()?

There are several possibilites to evaluate a model: hist = model.fit(x_train, y_train, (...) validation_data=(x_test, y_test)) or to use <...