29
votes
Accepted
What is purpose of the [CLS] token and why is its encoding output important?
CLS stands for classification and its there to represent sentence-level classification.
In short in order to make pooling scheme of BERT work this tag was introduced. I suggest reading up on this blog ...
- 5,529
21
votes
What is purpose of the [CLS] token and why is its encoding output important?
[CLS] stands for classification. It is added at the beginning because the training tasks here is sentence classification. And because they need an input that can represent the meaning of the entire ...
- 311
16
votes
What is purpose of the [CLS] token and why is its encoding output important?
In order to better understand the role of [CLS] let's recall that BERT model has been trained on 2 main tasks:
Masked language modeling: some random words are masked with [MASK]
token, the model ...
- 261
13
votes
What is purpose of the [CLS] token and why is its encoding output important?
Here're my understandings:
(1)[CLS] appears at the very beginning of each sentence, it has a fixed embedding and a fix positional embedding, thus this token contains no information itself.
(2)However, ...
- 131
12
votes
How to automatically classify a sentence or text based on its context?
Me: Please give 2 semantic tags for the sentence "The area of a circle is pi time the radius squared"
ChatGPT: 1. Mathematics. 2. Geometry
I'm not sure it's a robust and scalable solution ...
- 331
10
votes
Accuracy is getting worse after text pre processing
There are multiple possible reasons.
First, accuracy is certainly not a good evaluation measure for a multiclass problem, unless the dataset is balanced but I doubt it. You should use precision/...
- 24.6k
9
votes
Overfitting with text classification using Transformers
Your model is overfitting. You should try standard methods people use to prevent overfitting:
Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) ...
- 1,631
9
votes
Accepted
Accuracy is getting worse after text pre processing
You have to apply the same preprocessing to the test data.
Based on your code you apply the clean_text function only to train data but then predict on test/validation data that was not cleaned. That ...
- 106
9
votes
Accepted
How to automatically classify a sentence or text based on its context?
To my knowledge, there is no such library or pre-trained model.
Imho there is an important issue in the task as defined in the question, more exactly in the example: these tags seem natural for a ...
- 24.6k
6
votes
Accepted
How to preprocess with NLP a big dataset for text classification
Let me first clarify the general principle of classification with text data. Note that I'm assuming that you're using a "traditional" method (like decision trees), as opposed to Deep ...
- 24.6k
5
votes
How to use ndcg metric for binary relevance
The nDCG depends on the relevance of each document as you can see on the Wikipedia definition. I guess you could use 0 and 1 as relevance scores, but then all relevant documents would have the same ...
- 51
5
votes
Accepted
Over-sampling: is my model over-fitting?
In order to get accurate results, you should not oversample the test set! Otherwise you are simply evaluating on synthetic samples that you yourself have created. The support on your classification ...
- 7,788
5
votes
Accepted
Effect of Stop-Word Removal on Transformers for Text Classification
Very interesting question.
Easy, but probably lazy answer
When using pre-trained models, it is always advised to feed it data similar to what it was trained with. Basically, if it matters, don't ...
- 5,336
4
votes
Accepted
Using Trainable=True in Keras Embedding obtained better performance
It depends upon where do you want to submit your results that you claim, and what is the submission criteria.
First, it is unclear if "lower error in regression" is training or validation/...
- 308
4
votes
Accepted
How many layers should I replace in transfer learning CNN
To build on the previous answer:
In transfer learning, the goal is to use a pre-trained model and tweak the model to then specialise it to suit a certain task.
So, what we do is, as SrJ has eluded to, ...
- 1,398
4
votes
Accepted
How to include categorical fields to enhance a text classification
Scikit-learn has compose.ColumnTransformer which
allows different columns or column subsets of the input to be
transformed separately and the features generated by each transformer
will be ...
- 19.8k
4
votes
In Text Classification if I get similar performance with 100 features and 200 features, which model should I go ahead with?
You should go with the simpler model, the one that needs fewer features. Fewer features means quicker training cycles, better interpretibility and a faster forward pass. All of these are important ...
- 775
3
votes
Text classification based on n-grams and similarity
First of all, both in your question and your tags, you used Clustering and Classification interchangeably. Be careful as they are totally different problems. I give you a comprehensive tour on how to ...
- 6,472
3
votes
Accepted
k-means and LDA for text classification: how to test accuracy?
First of all, you use two terms Clustering and Classification interchangably and I would like to draw your attention to this. Your problem is purely Clustering.
Secondly, you asked for testing ...
- 6,472
3
votes
Accepted
Sentence embeddings with LSTM to classify the sentences is not working
The problem is that you are encoding the pieces of text as vectors and feeding those vectors to the model, but then the first layer of the model is again an embedding layer.
You should only use the ...
- 20.5k
3
votes
Performing a text classification based on a dictionary
As defined, there's no ML in this problem: the program would associate each keyword to a category, so it would consist of a loop over the words in the documents, and inside the loop there is an if ...
- 24.6k
3
votes
Accepted
Topic classification on text data with no/few labels
A feasible approach would be to take a pre-trained model, like BERT, and fine-tune it on a small labeled dataset.
For that, you may use Huggingface's Transformers, which makes all the steps in the ...
- 20.5k
3
votes
Is it valid changing the classification treshold of neural networks for improving the classification performance?
Yes it is a very common thing to do, for controlling tradeoff of objectives.
One often encountered example is to precision-recall tradeoff where we move the threshold to strike a balance between ...
- 714
2
votes
How to handle such a large class imbalance in text data?
For the most part, as you add more classes to your multiclass classification problem, it becomes more difficult to construct a model. All algorithms can run into trouble because there are fewer ...
- 121
2
votes
Text classification into thousands of classes
In general this doesn't work well, since it's almost unavoidable that the classifier won't be able to distinguish all the categories from each other. I'd suggest tying to reduce the number of ...
- 24.6k
2
votes
Accepted
Which insights a data scientist could derive from text-analysis?
Well, obviously the use cases depends on the industry. Also, I am assuming you are thinking of use cases that are somehow useful. But let's think of some examples:
I once worked with a book ...
- 335
2
votes
Accepted
Classify text as logical/ not logical
What you need is simply a language model. This is a very common task so you should be able to find code and data easily. This question gives some pointers for Python (be careful, the accepted answer ...
- 24.6k
2
votes
Accepted
Text Classification : Classifying N classes vs rest of the classes
This is called an open-class text classification problem, it's used in particular for some author identification problems.
I don't have any recent pointers but from a quick search I found this article:...
- 24.6k
2
votes
use genetic algorithm as a feature selection for text classification
There's a python library that helps do this task. TextFeatureSelection is the library and TextFeatureSelectionGA is the module.
https://pypi.org/project/TextFeatureSelection/
- 865
2
votes
Accepted
Sampling in Text Classification: can the results be considered 'reliable'?
There seems to be a mistake in your method:
I read about the use of downsampling and upsampling, so I applied them before training and testing the dataset.
It's incorrect to change the distribution ...
- 24.6k
Only top scored, non community-wiki answers of a minimum length are eligible