34 votes
Accepted

What is purpose of the [CLS] token and why is its encoding output important?

CLS stands for classification and its there to represent sentence-level classification. In short in order to make pooling scheme of BERT work this tag was introduced. I suggest reading up on this blog ...
Noah Weber's user avatar
  • 5,669
24 votes

What is purpose of the [CLS] token and why is its encoding output important?

[CLS] stands for classification. It is added at the beginning because the training tasks here is sentence classification. And because they need an input that can represent the meaning of the entire ...
Malgo's user avatar
  • 341
19 votes

What is purpose of the [CLS] token and why is its encoding output important?

In order to better understand the role of [CLS] let's recall that BERT model has been trained on 2 main tasks: Masked language modeling: some random words are masked with [MASK] token, the model ...
hoang tran's user avatar
15 votes

What is purpose of the [CLS] token and why is its encoding output important?

Here're my understandings: (1)[CLS] appears at the very beginning of each sentence, it has a fixed embedding and a fix positional embedding, thus this token contains no information itself. (2)However, ...
BigMoyan's user avatar
  • 151
12 votes

How to automatically classify a sentence or text based on its context?

Me: Please give 2 semantic tags for the sentence "The area of a circle is pi time the radius squared" ChatGPT: 1. Mathematics. 2. Geometry I'm not sure it's a robust and scalable solution ...
Sergey Skripko's user avatar
10 votes

Overfitting with text classification using Transformers

Your model is overfitting. You should try standard methods people use to prevent overfitting: Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) ...
Jindřich's user avatar
  • 1,741
10 votes

Accuracy is getting worse after text pre processing

There are multiple possible reasons. First, accuracy is certainly not a good evaluation measure for a multiclass problem, unless the dataset is balanced but I doubt it. You should use precision/...
Erwan's user avatar
  • 25.2k
9 votes
Accepted

Accuracy is getting worse after text pre processing

You have to apply the same preprocessing to the test data. Based on your code you apply the clean_text function only to train data but then predict on test/validation data that was not cleaned. That ...
aEmQy01b's user avatar
  • 106
9 votes
Accepted

How to automatically classify a sentence or text based on its context?

To my knowledge, there is no such library or pre-trained model. Imho there is an important issue in the task as defined in the question, more exactly in the example: these tags seem natural for a ...
Erwan's user avatar
  • 25.2k
6 votes
Accepted

Effect of Stop-Word Removal on Transformers for Text Classification

Very interesting question. Easy, but probably lazy answer When using pre-trained models, it is always advised to feed it data similar to what it was trained with. Basically, if it matters, don't ...
Valentin Calomme's user avatar
6 votes
Accepted

How to preprocess with NLP a big dataset for text classification

Let me first clarify the general principle of classification with text data. Note that I'm assuming that you're using a "traditional" method (like decision trees), as opposed to Deep ...
Erwan's user avatar
  • 25.2k
5 votes

How to use ndcg metric for binary relevance

The nDCG depends on the relevance of each document as you can see on the Wikipedia definition. I guess you could use 0 and 1 as relevance scores, but then all relevant documents would have the same ...
ovpira's user avatar
  • 51
5 votes
Accepted

Over-sampling: is my model over-fitting?

In order to get accurate results, you should not oversample the test set! Otherwise you are simply evaluating on synthetic samples that you yourself have created. The support on your classification ...
Djib2011's user avatar
  • 7,948
4 votes
Accepted

Using Trainable=True in Keras Embedding obtained better performance

It depends upon where do you want to submit your results that you claim, and what is the submission criteria. First, it is unclear if "lower error in regression" is training or validation/...
Emil's user avatar
  • 308
4 votes
Accepted

How many layers should I replace in transfer learning CNN

To build on the previous answer: In transfer learning, the goal is to use a pre-trained model and tweak the model to then specialise it to suit a certain task. So, what we do is, as SrJ has eluded to, ...
shepan6's user avatar
  • 1,428
4 votes
Accepted

How to include categorical fields to enhance a text classification

Scikit-learn has compose.ColumnTransformer which allows different columns or column subsets of the input to be transformed separately and the features generated by each transformer will be ...
Brian Spiering's user avatar
4 votes

In Text Classification if I get similar performance with 100 features and 200 features, which model should I go ahead with?

You should go with the simpler model, the one that needs fewer features. Fewer features means quicker training cycles, better interpretibility and a faster forward pass. All of these are important ...
Jayaram Iyer's user avatar
3 votes

Text classification based on n-grams and similarity

First of all, both in your question and your tags, you used Clustering and Classification interchangeably. Be careful as they are totally different problems. I give you a comprehensive tour on how to ...
Kasra Manshaei's user avatar
3 votes
Accepted

k-means and LDA for text classification: how to test accuracy?

First of all, you use two terms Clustering and Classification interchangably and I would like to draw your attention to this. Your problem is purely Clustering. Secondly, you asked for testing ...
Kasra Manshaei's user avatar
3 votes
Accepted

Sentence embeddings with LSTM to classify the sentences is not working

The problem is that you are encoding the pieces of text as vectors and feeding those vectors to the model, but then the first layer of the model is again an embedding layer. You should only use the ...
noe's user avatar
  • 25.6k
3 votes

Performing a text classification based on a dictionary

As defined, there's no ML in this problem: the program would associate each keyword to a category, so it would consist of a loop over the words in the documents, and inside the loop there is an if ...
Erwan's user avatar
  • 25.2k
3 votes
Accepted

Topic classification on text data with no/few labels

A feasible approach would be to take a pre-trained model, like BERT, and fine-tune it on a small labeled dataset. For that, you may use Huggingface's Transformers, which makes all the steps in the ...
noe's user avatar
  • 25.6k
3 votes

Is it valid changing the classification treshold of neural networks for improving the classification performance?

Yes it is a very common thing to do, for controlling tradeoff of objectives. One often encountered example is to precision-recall tradeoff where we move the threshold to strike a balance between ...
lpounng's user avatar
  • 998
2 votes

How to handle such a large class imbalance in text data?

For the most part, as you add more classes to your multiclass classification problem, it becomes more difficult to construct a model. All algorithms can run into trouble because there are fewer ...
thomaskolasa's user avatar
2 votes

Text classification into thousands of classes

In general this doesn't work well, since it's almost unavoidable that the classifier won't be able to distinguish all the categories from each other. I'd suggest tying to reduce the number of ...
Erwan's user avatar
  • 25.2k
2 votes
Accepted

Which insights a data scientist could derive from text-analysis?

Well, obviously the use cases depends on the industry. Also, I am assuming you are thinking of use cases that are somehow useful. But let's think of some examples: I once worked with a book ...
Guillermo Mosse's user avatar
2 votes
Accepted

Classify text as logical/ not logical

What you need is simply a language model. This is a very common task so you should be able to find code and data easily. This question gives some pointers for Python (be careful, the accepted answer ...
Erwan's user avatar
  • 25.2k
2 votes
Accepted

Text Classification : Classifying N classes vs rest of the classes

This is called an open-class text classification problem, it's used in particular for some author identification problems. I don't have any recent pointers but from a quick search I found this article:...
Erwan's user avatar
  • 25.2k
2 votes

use genetic algorithm as a feature selection for text classification

There's a python library that helps do this task. TextFeatureSelection is the library and TextFeatureSelectionGA is the module. https://pypi.org/project/TextFeatureSelection/
StatguyUser's user avatar
2 votes
Accepted

Sampling in Text Classification: can the results be considered 'reliable'?

There seems to be a mistake in your method: I read about the use of downsampling and upsampling, so I applied them before training and testing the dataset. It's incorrect to change the distribution ...
Erwan's user avatar
  • 25.2k

Only top scored, non community-wiki answers of a minimum length are eligible