37
votes
Accepted
What is purpose of the [CLS] token and why is its encoding output important?
CLS stands for classification and its there to represent sentence-level classification.
In short in order to make pooling scheme of BERT work this tag was introduced. I suggest reading up on this blog ...
- 5,759
30
votes
Accepted
NLP - why is "not" a stop word?
Stop words are usually thought of as "the most common words in a language". However, other definitions based on different tasks are possible.
It clearly makes sense to consider 'not' as a stop word ...
- 6,452
26
votes
What is purpose of the [CLS] token and why is its encoding output important?
[CLS] stands for classification. It is added at the beginning because the training tasks here is sentence classification. And because they need an input that can represent the meaning of the entire ...
- 361
19
votes
What is purpose of the [CLS] token and why is its encoding output important?
In order to better understand the role of [CLS] let's recall that BERT model has been trained on 2 main tasks:
Masked language modeling: some random words are masked with [MASK]
token, the model ...
- 291
15
votes
What is purpose of the [CLS] token and why is its encoding output important?
Here're my understandings:
(1)[CLS] appears at the very beginning of each sentence, it has a fixed embedding and a fix positional embedding, thus this token contains no information itself.
(2)However, ...
- 151
10
votes
Sentiment retriving from text (Russian)
Take a look at the polyglot library. It has polarity lexicons for 136 languages, including Russian.
The scale of the words’ polarity consisted of three degrees: +1 for
positive words, and -1 for ...
- 101
9
votes
Accepted
What is parts of speech technique in sentiment analysis?
Parts of Speech (POS)
This is what it is called when you label each of the words (often called tokens) of a sentence or many sentences. Usually they are labelled with grammatical descriptions, such ...
- 15.1k
7
votes
How to overcome training example's different lengths when working with Word Embeddings (word2vec)
Let me suggest three simple options:
average the vectors (component-wise), i.e., compute the word embedding vector for each word in the text, and average them. (as suggested by others).
take the (...
- 3,401
7
votes
Improving accuracy of Text Classification
First of all good job done in processing the data and coming up with your base model. I would suggest few things that you can try:
Improve your model my adding bigrams and tri-grams as features.
Try ...
7
votes
Accepted
Why using a frozen embedding layer in an LSTM model
The embedding matrix which used in the initialization of the Embedding layer is highly trained on a large corpus of text. The training and the data are so huge that ...
- 2,220
6
votes
BPE vs WordPiece Tokenization - when to use / which?
(This answer was originally a comment)
You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end ...
- 27.5k
5
votes
Accepted
Range to define emotions
The final range of emotion is completely arbitrary. No matter the interval [a, b], you can adjust the emotions to fit inside. [-100, 100] is perfectly reasonable and is common. An example of use is ...
- 206
5
votes
How to add more features in addition to a 100D word vector
You can add a second Input layer to your architecture.
Look at this link Keras Guide Multiple Inputs for a more detailed explanation.
The output of the embedding layer can be combined with the ...
- 773
5
votes
Accepted
How does one go about feature extraction for training labelled tweets for sentiment analysis?
Feature Extraction is an important step when dealing with natural languages because the text you've collected isn't in a form understandable by a computer. If you have a tweet that goes something like
...
5
votes
Accepted
NLP - How to perform semantic analysis?
With your three labels: positive, neutral or negative - it seems you are talking more about sentiment analysis. This answer the question: what are the emotions of the person who wrote this piece of ...
- 15.1k
5
votes
BPE vs WordPiece Tokenization - when to use / which?
Adding more info to noe's answer:
The difference between BPE and WordPiece lies in the way the symbol pairs are chosen for adding to the vocabulary. Instead of relying on the frequency of the pairs, ...
- 151
5
votes
Binary classification and numerical labels
If you’re going to have more than two labels, you need to go with a softmax activation and a loss for multi class classification, ie cross entropy loss.
Also, be cautious for multi-class versus multi-...
- 2,078
4
votes
Accepted
Understanding of naive bayes: computing the conditional probabilities
Your formula is correct for one $w_i$, but if you want to classify a document, you need to compute $P(c | w_1,\ldots,w_N)$.
Then you have $$P(c | w_1,\ldots,w_N) = \frac{P(c)\cdot P(w_1,\ldots,w_N|c)}...
- 6,452
4
votes
Accepted
Naive Bayes for SA in Scikit Learn - how does it work
Collecting your data
From the comments you state that you wish to classify comments into a label (1-poor, 2-fair, 3-ok, 4-good, 5-very good). Thus you will be training a model that maps a set of ...
- 9,056
4
votes
Accepted
Using Trainable=True in Keras Embedding obtained better performance
It depends upon where do you want to submit your results that you claim, and what is the submission criteria.
First, it is unclear if "lower error in regression" is training or validation/...
- 308
4
votes
How is it possible for RNN to do sentiment analysis?
RNNs do not learn to predict sentiment. They learn correlations between the input data and the target labels. If they see that every time the input contains the word "bad" they have to ...
- 27.5k
3
votes
Accepted
How do you map phrase IDs to sentence IDs in the Stanford Sentiment Analysis dataset?
Only full sentences are used for testing and validation, though sentences and phrases are used for training. See Yoon Kim 2014, Convolutional Neural Networks for Sentence Classification for an ...
- 181
3
votes
Accepted
Understand clearly the figure: Illustration of a Convolutional Neural Network (CNN) architecture for sentence classification
Answering this in terms of NLP examples is quite hard, remember "All models are wrong, some models are useful." First think of this in an image classification problem context, you want to use a large ...
- 1,858
3
votes
How to implement multi class classifier for a set of sentences?
I was thinking if Apriori might be more suited for your purpose.
For your consideration:
1) Tokenize the training sentences into bag of words:
Review Sentence | Upside | Calm | Swimming | Cat
2) Tag ...
- 462
3
votes
What is the current state-of-the-art within aspect-based sentiment analysis?
It's hard to say what is state-of-the-art in general without breaking aspect-level sentiment analysis down into its subtasks:
1) Aspect extraction
2) Sentiment classification
As you've probably ...
- 31
3
votes
Accepted
Words to numbers faster lookup
I'd like to extend the great @Emre's answer with another example - we are going to replace all tokenized words from the "1984" (c) George Orwell (120K words):
...
3
votes
Accepted
Best way to fix the size of a sentence [Sentiment Analysis]
The easiest way is to average the word- embeddings. This works quite well.
Another thing you can try is to represent each document as a bag of words - i.e. - to have a vector in the size of your ...
- 225
3
votes
What is parts of speech technique in sentiment analysis?
Parts of Speech explains how a word is used in a sentence, i.e whether it is a verb, noun, adjective and so on.
In text processing, those POS (or word classes) are usually represented as their ...
- 836
3
votes
Accepted
any efficient way to find surrounding adjective/verbs with respect to the target phrase in python [updated]?
POS-tagging consist of qualifying words by attaching a Part-Of-Speech to it. Part-Of-Speech is a tag that indicates the role of a word in a sentence (e.g. a noun, a transitive verb, a comparative ...
- 267
3
votes
Accepted
Differentiate between positive and negative clusters
Maybe you don't have a positive and a negative class.
Your input are word vectors. Unless you trained your word vectors before with explicit positive and negative labels, it is very unlikely that your ...
- 3,608
Only top scored, non community-wiki answers of a minimum length are eligible