34 votes
Accepted

What is purpose of the [CLS] token and why is its encoding output important?

CLS stands for classification and its there to represent sentence-level classification. In short in order to make pooling scheme of BERT work this tag was introduced. I suggest reading up on this blog ...
Noah Weber's user avatar
  • 5,669
29 votes
Accepted

NLP - why is "not" a stop word?

Stop words are usually thought of as "the most common words in a language". However, other definitions based on different tasks are possible. It clearly makes sense to consider 'not' as a stop word ...
oW_'s user avatar
  • 6,337
24 votes

What is purpose of the [CLS] token and why is its encoding output important?

[CLS] stands for classification. It is added at the beginning because the training tasks here is sentence classification. And because they need an input that can represent the meaning of the entire ...
Malgo's user avatar
  • 341
19 votes

What is purpose of the [CLS] token and why is its encoding output important?

In order to better understand the role of [CLS] let's recall that BERT model has been trained on 2 main tasks: Masked language modeling: some random words are masked with [MASK] token, the model ...
hoang tran's user avatar
15 votes

What is purpose of the [CLS] token and why is its encoding output important?

Here're my understandings: (1)[CLS] appears at the very beginning of each sentence, it has a fixed embedding and a fix positional embedding, thus this token contains no information itself. (2)However, ...
BigMoyan's user avatar
  • 151
10 votes

Sentiment retriving from text (Russian)

Take a look at the polyglot library. It has polarity lexicons for 136 languages, including Russian. The scale of the words’ polarity consisted of three degrees: +1 for positive words, and -1 for ...
vrs's user avatar
  • 101
10 votes
Accepted

Features of word vectors in Word2Vec

1- The number of features: In terms of neural network model it represents the number of neurons in the projection(hidden) layer. As the projection layer is built upon distributional hypothesis, ...
chmodsss's user avatar
  • 1,964
9 votes
Accepted

What is parts of speech technique in sentiment analysis?

Parts of Speech (POS) This is what it is called when you label each of the words (often called tokens) of a sentence or many sentences. Usually they are labelled with grammatical descriptions, such ...
n1k31t4's user avatar
  • 14.8k
8 votes

Training Dataset for Sentiment Analysis of Movie Reviews

You can use the SAR14 dataset of 234K IMDb movie reviews. The construction of the SAR14 dataset is detailed in the paper "Sentiment Classification on Polarity Reviews: An Empirical Study Using Rating-...
NQD's user avatar
  • 211
7 votes

How to overcome training example's different lengths when working with Word Embeddings (word2vec)

Let me suggest three simple options: average the vectors (component-wise), i.e., compute the word embedding vector for each word in the text, and average them. (as suggested by others). take the (...
D.W.'s user avatar
  • 3,341
7 votes

Improving accuracy of Text Classification

First of all good job done in processing the data and coming up with your base model. I would suggest few things that you can try: Improve your model my adding bigrams and tri-grams as features. Try ...
Santanu_Pattanayak's user avatar
7 votes
Accepted

Why using a frozen embedding layer in an LSTM model

The embedding matrix which used in the initialization of the Embedding layer is highly trained on a large corpus of text. The training and the data are so huge that ...
Shubham Panchal's user avatar
6 votes
Accepted

Sentiment retriving from text (Russian)

If you are seeking a working solution, I know of an API that supports many languages, including Russian: indico.io Text Analysis sentiment() ...
Adam Bittlingmayer's user avatar
6 votes

BPE vs WordPiece Tokenization - when to use / which?

(This answer was originally a comment) You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end ...
noe's user avatar
  • 25.7k
5 votes

How do I assess which sentiment classifier is best for my project?

A couple of important points: Sentiment analysis is not an exact science. Two people, reading the same text in different contexts will come to different conclusions about sentiment, especially on ...
Neil Slater's user avatar
  • 28.9k
5 votes
Accepted

Range to define emotions

The final range of emotion is completely arbitrary. No matter the interval [a, b], you can adjust the emotions to fit inside. [-100, 100] is perfectly reasonable and is common. An example of use is ...
Stefan G.'s user avatar
  • 206
5 votes

How to add more features in addition to a 100D word vector

You can add a second Input layer to your architecture. Look at this link Keras Guide Multiple Inputs for a more detailed explanation. The output of the embedding layer can be combined with the ...
Nanda's user avatar
  • 773
5 votes
Accepted

How does one go about feature extraction for training labelled tweets for sentiment analysis?

Feature Extraction is an important step when dealing with natural languages because the text you've collected isn't in a form understandable by a computer. If you have a tweet that goes something like ...
edthealchemist's user avatar
5 votes
Accepted

NLP - How to perform semantic analysis?

With your three labels: positive, neutral or negative - it seems you are talking more about sentiment analysis. This answer the question: what are the emotions of the person who wrote this piece of ...
n1k31t4's user avatar
  • 14.8k
5 votes

Binary classification and numerical labels

If you’re going to have more than two labels, you need to go with a softmax activation and a loss for multi class classification, ie cross entropy loss. Also, be cautious for multi-class versus multi-...
hH1sG0n3's user avatar
  • 2,008
4 votes

How to overcome training example's different lengths when working with Word Embeddings (word2vec)

Two very different suggestions here to avoid averging the vectors: Use Word Mover's Distance (https://github.com/mkusner/wmd) to compute distance between the tweets (not sure how well it would work ...
Sharon's user avatar
  • 49
4 votes

How do I assess which sentiment classifier is best for my project?

You can classify a few of the tweets yourself, and compare afterwards which of the two algorithmic results is closer to your classification. Without more information we cannot tell what these ...
knb's user avatar
  • 602
4 votes
Accepted

Understanding of naive bayes: computing the conditional probabilities

Your formula is correct for one $w_i$, but if you want to classify a document, you need to compute $P(c | w_1,\ldots,w_N)$. Then you have $$P(c | w_1,\ldots,w_N) = \frac{P(c)\cdot P(w_1,\ldots,w_N|c)}...
oW_'s user avatar
  • 6,337
4 votes
Accepted

Naive Bayes for SA in Scikit Learn - how does it work

Collecting your data From the comments you state that you wish to classify comments into a label (1-poor, 2-fair, 3-ok, 4-good, 5-very good). Thus you will be training a model that maps a set of ...
JahKnows's user avatar
  • 8,856
4 votes
Accepted

Using Trainable=True in Keras Embedding obtained better performance

It depends upon where do you want to submit your results that you claim, and what is the submission criteria. First, it is unclear if "lower error in regression" is training or validation/...
Emil's user avatar
  • 308
4 votes

BPE vs WordPiece Tokenization - when to use / which?

Adding more info to noe's answer: The difference between BPE and WordPiece lies in the way the symbol pairs are chosen for adding to the vocabulary. Instead of relying on the frequency of the pairs, ...
Abhi25t's user avatar
  • 141
4 votes

How is it possible for RNN to do sentiment analysis?

RNNs do not learn to predict sentiment. They learn correlations between the input data and the target labels. If they see that every time the input contains the word "bad" they have to ...
noe's user avatar
  • 25.7k
3 votes

Training Dataset for Sentiment Analysis of Movie Reviews

There are many datasets available. Multi-Domain Sentiment Dataset Twitter sentiment UCI Sentiment Analysis Dataset Large Movie Review Dataset
Sreejithc321's user avatar
  • 1,920
3 votes

Twitter Sentiment Analysis: Detecting neutral tweets despite training on only Positive and Negative Classes

The quick (and not very satisfying) answer is "it depends" -- specifically it depends upon what your underlying conceptual model of human emotion is and how it manifests in verbal/written behavior. ...
Brandon Loudermilk's user avatar
3 votes

How to implement multi class classifier for a set of sentences?

I was thinking if Apriori might be more suited for your purpose. For your consideration: 1) Tokenize the training sentences into bag of words: Review Sentence | Upside | Calm | Swimming | Cat 2) Tag ...
jkyh's user avatar
  • 462

Only top scored, non community-wiki answers of a minimum length are eligible