7

CLS stands for classification and its there to represent sentence-level classification. In short in order to make pooling scheme of BERT work this tag was introduced. I suggest reading up on this blog where this is also covered in detail.


6

[CLS] stands for classification. It is added at the beginning because the training tasks here is sentence classification. And because they need an input that can represent the meaning of the entire sentence, they introduce a new tag. They can’t take any other word from the input sequence, because the output of that is the word representation. So they add a ...


5

Your model is overfitting. You should try standard methods people use to prevent overfitting: Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) also sometimes help (0.1-0.3 might be reasonable values). If you have many input classes, label smoothing can help. You can try a smaller model dimension. If you ...


5

In order to get accurate results, you should not oversample the test set! Otherwise you are simply evaluating on synthetic samples that you yourself have created. The support on your classification report should mirror the imbalance in your dataset. From what I understand you have 3500 samples, then you did some oversampling (probably brought them to around ...


5

Let me first clarify the general principle of classification with text data. Note that I'm assuming that you're using a "traditional" method (like decision trees), as opposed to Deep Learning (DL) method. As you correctly understand, each individual text document (instance) has to be represented as a vector of features, each feature representing a ...


4

Scikit-learn has compose.ColumnTransformer which allows different columns or column subsets of the input to be transformed separately and the features generated by each transformer will be concatenated to form a single feature space. This is useful for heterogeneous or columnar data, to combine several feature extraction mechanisms or transformations into a ...


3

To build on the previous answer: In transfer learning, the goal is to use a pre-trained model and tweak the model to then specialise it to suit a certain task. So, what we do is, as SrJ has eluded to, keep the main model's architecture in tact. So this would be the 6 CNN layers (and possibly the three linear layers, if they were also involved in pre-training)...


3

The nDCG depends on the relevance of each document as you can see on the Wikipedia definition. I guess you could use 0 and 1 as relevance scores, but then all relevant documents would have the same score of 1, and then it wouldn't make much sense to apply the nDCG penalty discounts. A similar measure often used with binary relevance scores is the mean ...


2

In general this doesn't work well, since it's almost unavoidable that the classifier won't be able to distinguish all the categories from each other. I'd suggest tying to reduce the number of categories (for instance discard the least common ones). In any case I'm not aware of any specific model to deal with a high number of classes, it's regular text ...


2

Well, obviously the use cases depends on the industry. Also, I am assuming you are thinking of use cases that are somehow useful. But let's think of some examples: I once worked with a book distributor that tagged each book they sold with keywords (Fantasy, Horror, etc). You can automate the tagging process if you have a sufficiently large dataset of ...


2

First of all, you use two terms Clustering and Classification interchangably and I would like to draw your attention to this. Your problem is purely Clustering. Secondly, you asked for testing accuracy. As your problem is pure Clustering, there is no evaluation for that. The last but not least is the problem of "Short Text Understanding". In short ...


2

What you need is simply a language model. This is a very common task so you should be able to find code and data easily. This question gives some pointers for Python (be careful, the accepted answer is incorrect according to the two other answers). Applying the language model to a sentence gives you a probability (or a perplexity score, which works the ...


2

This is called an open-class text classification problem, it's used in particular for some author identification problems. I don't have any recent pointers but from a quick search I found this article: https://www.aclweb.org/anthology/N16-1061.pdf In the field of author classification there is a similar problem called author verification, which can be ...


2

There seems to be a mistake in your method: I read about the use of downsampling and upsampling, so I applied them before training and testing the dataset. It's incorrect to change the distribution of the test set. When resampling, the resampling should be applied only on the training set. The goal is to force the model to take into account the two classes,...


2

You could train a character-level language model, e.g. an LSTM, on the real short texts, and use the perplexity as the signal to know whether a piece of text is real or not. In order to find an appropriate perplexity threshold, you can have a look at the distribution of perplexities over a validation holdout dataset. UPDATE: There are multiple ...


2

What you are proposing is a heuristic method, because you define the rules manually in advance. From a Machine Learning (ML) point of view the "training" is the part where you observe some data and decide which rules to apply, and the "testing" is when you run a program which applies these rules to obtain a predicted label. As you ...


2

First, congratulations for thinking to do a qualitative analysis of the results :) I know it should be obvious, but so many people just assume that the system works and don't bother checking their output. Now, strictly speaking what you're seeing is not a bug. These are errors made by a statistical system. A statistical system is not meant to get everything ...


2

In order to detect overfitting you need to separate your data in a training set - that you use to estimate the parameters of you model - and a test set - where you evaluate your model keeping the parameters fixed, this is usually called cross-validation. I understand from your results that you are not doing such separation of data so, you're not able to ...


2

Very interesting question. Easy, but probably lazy answer When using pre-trained models, it is always advised to feed it data similar to what it was trained with. Basically, if it matters, don't remove them, and if it doesn't matter, it doesn't hurt to keep them in. Obviously, if you can, try with or without stopwords, and see what works best for your ...


1

In neural networks meant for classification, you need a linear layer before the softmax to project the internal representation, which has some dimensionality $d_i$, to the output space, which has dimensionality $d_o$ equal the number of choices (5 in your case). So you either place a Dense(5) layer after the BiLSTM or you take the output of the BiLSTM "...


1

Accuracy is not the best measure for imbalanced data. Prefer precision and recall. Do undersampling/oversampling to get equal samples for each class and try XGBoost. Or else you can use SVC with class weights, give lower class weight to classes with more samples and vice versa.


1

Try to use the concatenated vector of average word2vec and max pool word2vec. You may get better results than tfidf and PCA. Remove the words like stop words. Also, try with lemmatized words. If you okay with computational time, go for transformer-based transfer learning embedding like BERT, ALBERT etc...


1

You have tagged this as text-classification, but describe your efforts to cluster the documents. My answer is based on the assumption that you don't know how the documents are grouped for you to classify new documents. The first step would be to determine the clusters/classes/groups and then classify new documents. There are at least 2 things one can think ...


1

It can be improved in many ways. However, the best start would be if you elaborate on your pipeline together with choosen parameters, as well as your data.


1

There are different ways to address the task that you describe: If the goal is simply to predict the author among a set of predefined authors, then this is not a clustering task (unsupervised) but a classification task (supervised). This implies that you would split the data between training and test set (or use cross-validation), train a model using the ...


1

Okay so keeping it very short and precisely in context of your question- Accuracy tells us, out of all the documents how many are classified correctly. Precision tells us out of all documents which are predicted in a category, how often its correct. Uni -gram- "nasa", "is" "space" , "agency" bi-gram- "nasa is", "space agency" Now lets go over the ...


1

For the most part, as you add more classes to your multiclass classification problem, it becomes more difficult to construct a model. All algorithms can run into trouble because there are fewer examples of each class to learn from. In case the 27 class target feature does not work well for you, you can decrease the number of classes even more. To do this: ...


1

I too am working on same problem, found these below links very useful in getting started on oversampling and under sampling- https://machinelearningmastery.com/random-oversampling-and-undersampling-for-imbalanced-classification/ https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis


1

Regarding the first question, to the best of my knowledge, there is no SOTA approach. It depends on the task. For the second one, did you try to combine the numerical values with the TFIDF vectors, and then feed the final vector(s) to the classifier? Probably not. I usually try this way. But, if your vocabulary is too large, try to reduce it by considering ...


Only top voted, non community-wiki answers of a minimum length are eligible