204
Micro- and macro-averages (for whatever metric) will compute slightly different things, and thus their interpretation differs. A macro-average will compute the metric independently for each class and then take the average (hence treating all classes equally), whereas a micro-average will aggregate the contributions of all classes to compute the average ...
25
Original Post - http://rushdishams.blogspot.in/2011/08/micro-and-macro-average-of-precision.html
In Micro-average method, you sum up the individual true positives, false positives, and false negatives of the system for different sets and the apply them to get the statistics.
Tricky, but I found this very interesting. There are two methods by which you can ...
19
scale_pos_weight is used for binary classification as you stated. It is a more generalized solution to handle imbalanced classes. A good approach when assigning a value to scale_pos_weight is:
sum(negative instances) / sum(positive instances)
For your specific case, there is another option in order to weight individual data points and take their weights ...
14
In a multi-class setting micro-averaged precision and recall are always the same.
$$
P = \frac{\sum_c TP_c}{\sum_c TP_c + \sum_c FP_c}\\
R = \frac{\sum_c TP_c}{\sum_c TP_c + \sum_c FN_c}
$$
where c is the class label.
Since in a multi-class setting you count all false instances it turns out that
$$
\sum_c FP_c = \sum_c FN_c
$$
Hence P = R. In other words, ...
11
Always split before you do any data pre-processing. Performing pre-processing before splitting will mean that information from your test set will be present during training, causing a data leak.
Think of it like this, the test set is supposed to be a way of estimating performance on totally unseen data. If it affects the training, then it will be partially ...
9
I would not use the word "best" but LSTM-RNN are very powerful when dealing with timeseries, simply because they can store information about previous values and exploit the time dependencies between the samples. That said, it is definitely worth going for it.
It has been proven that their performance can be boosted significantly if they are combined with a ...
9
It sounds like you don't want to start retraining the model every time a new label category appears. The easiest way to retain maximal information of past data would be train one classifier per category.
This way you can continue to train each classifier incrementally ("online") with something like SGDClassifier without having to retrain them.
Whenever a ...
8
This answer by @KeremT is correct. I provide an example for those who still have problems with the exact implementation.
weight parameter in XGBoost is per instance not per class. Therefore, we need to assign the weight of each class to its instances, which is the same thing.
For example, if we have three imbalanced classes with ratios
class A = 10%
class ...
7
I would use the functional interface.
Something like this:
from keras.layers import Activation, Input, Dense
from keras.models import Model
from keras.layers.merge import Concatenate
input_ = Input(shape=input_shape)
x = input_
x1 = Dense(4, x)
x2 = Dense(4, x)
x3 = Dense(4, x)
x1 = Activation('softmax')(x1)
x2 = Activation('softmax')(x2)
x3 = Activation(...
6
Absolutely you can create an approach that forces high-precision class tagging algorithm (at the natural cost of recall). What's more- you can do this with (at least) any method that provides a percentage calue for predictions, which is the vast majority of classifiers. The key is, as you mention, to find the minimum acceptable value of precision and cut the ...
6
Nice question!
Some Remarks
For imbalanced data you have different approaches. Most well-established one is resampling (Oversampling small classes /underssampling large classes). The other one is to make your classification hierarchical i.e. classify large classes against all others and then classify small classes in second step (classifiers are not ...
6
1) A five-layer neural network is one heck of a complex model for a data set with less than 1 million points. (I’m trying to find a good link for this, but the intuition is that your choice of model should be driven by the complexity of the available data, and not by what you think the real target function is like.) If this is for a real-world project, a ...
5
You might find it useful to treat n-grams of characters as your feature space. Then you could represent a string as a bag of substrings. With N = 4 or greater, you would capture things like ".com" in emails, "##.#" in dates, etc.
It might also help to encode all single digits as one reserved number-only-character.
An easy way to to this might be to create ...
5
1) Max-Entropy(Logistic Regression) on TFIDF vectors is a good starting point for many NLP classification task.
2) Word2vec is definitely something worth trying and comparing to model 1. I would suggest using the Doc2Vec flavor for looking at sentences/paragraphs.
Quoc Le and Tomas Mikolov. Distributed Representations of Sentences and Documents. http://...
5
A class taxonomy should:
Serve the business needs
Be learnable
There is a potential tradeoff here. The more exact and specific the taxonomy, the more you'll know about the entities and you'll be able to server better the business needs. However, for a large taxonomy the classifier will have to model more complex rules, will have less samples for each case ...
5
Everyone stumbles upon this question when dealing with unbalanced multiclass classification problem using XGBoost in R. I did too!
I was looking for an example to better understand how to apply it. Invested almost an hour to find the link mentioned below. For all those who are looking for an example, here goes -
https://datascience.stackexchange.com/a/9493/...
5
It could be a bug in your code, problems with your training set (maybe you don't have the file format quite right), or some other implementation issue.
Are you sure you want to use a sigmoid activation function in your last layer? I would have expected that the normal approach would be to use a softmax as the last layer (so that you can treat the outputs ...
5
It is possible just implement your own softmax function. You can split a tensor to parts, then compute softmax separately per part and concatenate tensor parts:
def custom_softmax(t):
sh = K.shape(t)
partial_sm = []
for i in range(sh[1] // 4):
partial_sm.append(K.softmax(t[:, i*4:(i+1)*4]))
return K.concatenate(partial_sm)
...
5
The greater the number of output nodes the higher complexity you will add to your model. This means that given a fixed amount of data, a greater number of output nodes will lead to poorer results. I would use a ABCD vs. others strategy.
Instead of conditioning your model to learn the distributions of the class A, B, C and D separately you will combine them. ...
5
No, it is perfectly possible to train on multiple categories. What you need, though, is an exhaustive list of these categories (in supervised learning, that is).
Suppose you are trying to associate sentences with topics, and you have a list of possible topics topics = ['sports', 'soccer', 'politics'].
It sounds like your data look something like this:
...
5
If you are looking for just an alternative loss function:
Focal Loss has been shown on imagenet to help with this problem indeed.
Focal loss adds a modulating factor to cross entropy loss ensuring that the negative/majority class/easy decisions not over whelm the loss due to the minority/hard classes.
I would look into using that is it seems to be ...
5
If you have enough data and reasonable number of classes, you can definitely train your model. The grouping of words that you have done is similar to an approach called bag-of-words model. You can use that to build a classifier using Naive Bayes or SVM etc.
On a different note, you can also look at the KNN algorithm because it looks fit for your use case. ...
5
If new categories are arriving very rarely, I myself prefer the "one vs all" solution provided by @oW_. For each new category, you train a new model on X number of samples from new category (class 1), and X number of samples from the rest of categories (class 0).
However, if new categories are arriving frequently and you want to use a single shared model, ...
5
You should not expect class_weight parameters and SMOTE to give the exact same results because they are different methods.
Class weights directly modify the loss function by giving more (or less) penalty to the classes with more (or less) weight. In effect, one is basically sacrificing some ability to predict the lower weight class (the majority class for ...
5
For predictive power, in general, including both shouldn't be a problem. But there is a lot of nuance here.
Foremost, if predictive power isn't all you care about: if you're making statistical inferences, or care about explainability and feature importances, then including both can cause issues. Briefly, your model may split the importance of the underlying ...
4
I am working on Random Forest Classifier and this classifier has probability attribute in prediction i.e if you get the summary of predictions = model.transform(testData) as print(predictions) in PySpark you will get the probability of each labels, You can check the below code and output of the code:
from pyspark.sql import DataFrame
from pyspark import ...
4
The reason NB is called "Naive" is that is makes the assumption that the predictive variables are all independent.
This assumption usually skews the model scores (which, under the above naive assumption are unbiased probability estimates) towards 0 or 1.
In your case, e.g., the presence of words flower and petal indicate gardening category, but, because the ...
4
I think you are making things more confusing then they are.
Binary
In this case you have two possible outputs:
Obama = 1.
Not-Obama (who in this case can only be Romney) = 0.
Multi-Class
In this case you have k possible outputs, for example when k = 4:
k = 0: Obama
k = 1: Romney
k = 2: Clinton
k = 3: Bush
There are approaches to tackle multi-...
4
We would need more information on the prediction problem and the features to be able to give something more precise.
Anyhow, I am surprised no answer so far included all possible options since they aren't that many:
get rid of incomplete observations or features --- obviously, only viable if there are few incomplete cases since you lose too much ...
4
I know this isn't answering the question that you actually asked, but I suggest that you NOT generate data for your 'short text' categorization problem.
Generated data can work for certain cases when data scientists who are very familiar with an algorithm want to demonstrate a specific feature, but there is a hokeyness that may lead you astray as someone ...
Only top voted, non community-wiki answers of a minimum length are eligible
