361
Micro- and macro-averages (for whatever metric) will compute slightly different things, and thus their interpretation differs. A macro-average will compute the metric independently for each class and then take the average (hence treating all classes equally), whereas a micro-average will aggregate the contributions of all classes to compute the average ...
36
This is the Original Post.
In Micro-average method, you sum up the individual true positives, false positives, and false negatives of the system for different sets and the apply them to get the statistics.
Tricky, but I found this very interesting. There are two methods by which you can get such average statistic of information retrieval and classification.
...
25
scale_pos_weight is used for binary classification as you stated. It is a more generalized solution to handle imbalanced classes. A good approach when assigning a value to scale_pos_weight is:
sum(negative instances) / sum(positive instances)
For your specific case, there is another option in order to weight individual data points and take their weights ...
23
In a multi-class setting micro-averaged precision and recall are always the same.
$$
P = \frac{\sum_c TP_c}{\sum_c TP_c + \sum_c FP_c}\\
R = \frac{\sum_c TP_c}{\sum_c TP_c + \sum_c FN_c}
$$
where c is the class label.
Since in a multi-class setting you count all false instances it turns out that
$$
\sum_c FP_c = \sum_c FN_c
$$
Hence P = R. In other words, ...
17
This answer by @KeremT is correct. I provide an example for those who still have problems with the exact implementation.
weight parameter in XGBoost is per instance not per class. Therefore, we need to assign the weight of each class to its instances, which is the same thing.
For example, if we have three imbalanced classes with ratios
class A = 10%
class ...
14
For sklearn version < 0.19
Just assign each entry of your train data its class weight. First get the class weights with class_weight.compute_class_weight of sklearn then assign each row of the train data its appropriate weight.
I assume here that the train data has the column class containing the class number. I assumed also that there are nb_classes that ...
14
Always split before you do any data pre-processing. Performing pre-processing before splitting will mean that information from your test set will be present during training, causing a data leak.
Think of it like this, the test set is supposed to be a way of estimating performance on totally unseen data. If it affects the training, then it will be partially ...
13
I would not use the word "best" but LSTM-RNN are very powerful when dealing with timeseries, simply because they can store information about previous values and exploit the time dependencies between the samples. That said, it is definitely worth going for it.
It has been proven that their performance can be boosted significantly if they are ...
12
If you are looking for just an alternative loss function:
Focal Loss has been shown on imagenet to help with this problem indeed.
Focal loss adds a modulating factor to cross entropy loss ensuring that the negative/majority class/easy decisions not over whelm the loss due to the minority/hard classes.
I would look into using that is it seems to be ...
10
I would use the functional interface.
Something like this:
from keras.layers import Activation, Input, Dense
from keras.models import Model
from keras.layers.merge import Concatenate
input_ = Input(shape=input_shape)
x = input_
x1 = Dense(4, x)
x2 = Dense(4, x)
x3 = Dense(4, x)
x1 = Activation('softmax')(x1)
x2 = Activation('softmax')(x2)
x3 = Activation(...
10
You should not expect class_weight parameters and SMOTE to give the exact same results because they are different methods.
Class weights directly modify the loss function by giving more (or less) penalty to the classes with more (or less) weight. In effect, one is basically sacrificing some ability to predict the lower weight class (the majority class for ...
9
It sounds like you don't want to start retraining the model every time a new label category appears. The easiest way to retain maximal information of past data would be train one classifier per category.
This way you can continue to train each classifier incrementally ("online") with something like SGDClassifier without having to retrain them.
Whenever a ...
8
If your task is a kind of classification that the labels are mutually exclusive, each input just has one label, you have to use Softmax. If the inputs of your classification task have multiple labels for an input, your classes are not mutually exclusive and you can use Sigmoid for each output. For the former case, you should choose the output entry with the ...
7
Indeed, this is the standard interpretation of continuous classifier outputs, not only for neural networks, but for the more general case called Softmax Regression.
Thus, provided that you have used softmax activation on the final layer (in order, among other things, to ensure that your outputs indeed sum up to 1), you can interpret the continuous outputs ...
7
The greater the number of output nodes the higher complexity you will add to your model. This means that given a fixed amount of data, a greater number of output nodes will lead to poorer results. I would use a ABCD vs. others strategy.
Instead of conditioning your model to learn the distributions of the class A, B, C and D separately you will combine them. ...
7
1) A five-layer neural network is one heck of a complex model for a data set with less than 1 million points. (I’m trying to find a good link for this, but the intuition is that your choice of model should be driven by the complexity of the available data, and not by what you think the real target function is like.) If this is for a real-world project, a ...
6
Everyone stumbles upon this question when dealing with unbalanced multiclass classification problem using XGBoost in R. I did too!
I was looking for an example to better understand how to apply it. Invested almost an hour to find the link mentioned below. For all those who are looking for an example, here goes.
Thanks wacax
6
Absolutely you can create an approach that forces high-precision class tagging algorithm (at the natural cost of recall). What's more- you can do this with (at least) any method that provides a percentage calue for predictions, which is the vast majority of classifiers. The key is, as you mention, to find the minimum acceptable value of precision and cut the ...
6
Nice question!
Some Remarks
For imbalanced data you have different approaches. Most well-established one is resampling (Oversampling small classes /underssampling large classes). The other one is to make your classification hierarchical i.e. classify large classes against all others and then classify small classes in second step (classifiers are not ...
6
If you have enough data and reasonable number of classes, you can definitely train your model. The grouping of words that you have done is similar to an approach called bag-of-words model. You can use that to build a classifier using Naive Bayes or SVM etc.
On a different note, you can also look at the KNN algorithm because it looks fit for your use case. ...
5
You might find it useful to treat n-grams of characters as your feature space. Then you could represent a string as a bag of substrings. With N = 4 or greater, you would capture things like ".com" in emails, "##.#" in dates, etc.
It might also help to encode all single digits as one reserved number-only-character.
An easy way to to this might be to create ...
5
1) Max-Entropy(Logistic Regression) on TFIDF vectors is a good starting point for many NLP classification task.
2) Word2vec is definitely something worth trying and comparing to model 1. I would suggest using the Doc2Vec flavor for looking at sentences/paragraphs.
Quoc Le and Tomas Mikolov. Distributed Representations of Sentences and Documents. http://...
5
A class taxonomy should:
Serve the business needs
Be learnable
There is a potential tradeoff here. The more exact and specific the taxonomy, the more you'll know about the entities and you'll be able to server better the business needs. However, for a large taxonomy the classifier will have to model more complex rules, will have less samples for each case ...
5
That's how it should be. I had the same result for my research. It seemed weird at first. But precision and recall should be the same while micro-averaging the result of multi-class single-label classifier. This is because if you consider a misclassification c1=c2 (where c1 and c2 are 2 different classes), the misclassification is a false positive (fp) with ...
5
It could be a bug in your code, problems with your training set (maybe you don't have the file format quite right), or some other implementation issue.
Are you sure you want to use a sigmoid activation function in your last layer? I would have expected that the normal approach would be to use a softmax as the last layer (so that you can treat the outputs ...
5
You learn a lot by comparing to a naive model. A naive model is one without any features. As a default, it will always predict the most likely Target. Note that this is exactly what your model is doing. This indicates that the features are not helping with making a prediction. Have you done a basic distribution analysis to see what are the features ...
5
It is possible just implement your own softmax function. You can split a tensor to parts, then compute softmax separately per part and concatenate tensor parts:
def custom_softmax(t):
sh = K.shape(t)
partial_sm = []
for i in range(sh[1] // 4):
partial_sm.append(K.softmax(t[:, i*4:(i+1)*4]))
return K.concatenate(partial_sm)
...
5
No, it is perfectly possible to train on multiple categories. What you need, though, is an exhaustive list of these categories (in supervised learning, that is).
Suppose you are trying to associate sentences with topics, and you have a list of possible topics topics = ['sports', 'soccer', 'politics'].
It sounds like your data look something like this:
...
5
I think, and have done similar problem too, that this problem can be solved in this way:
1. Generate NGrams
2. Create 1 hot encoding matrix
3. Pass to Naive Bayes or Random forest
It would automatically count the words count (you can apply TFIDF too) and based on that weightage will be calculated.
Examples:
https://medium.com/data-from-the-trenches/text-...
5
If new categories are arriving very rarely, I myself prefer the "one vs all" solution provided by @oW_. For each new category, you train a new model on X number of samples from new category (class 1), and X number of samples from the rest of categories (class 0).
However, if new categories are arriving frequently and you want to use a single shared model, ...
Only top voted, non community-wiki answers of a minimum length are eligible