21

Stop words are usually thought of as "the most common words in a language". However, other definitions based on different tasks are possible. It clearly makes sense to consider 'not' as a stop word if your task is based on word frequencies (e.g. tf–idf analysis for document classification). If you're concerned with the context (e.g. sentiment analysis) ...


10

Most often serialization error in (Py)Spark means that some part of your distributed code (e.g. functions passed to map) has dependencies on non-serializable data. Consider following example: rdd = sc.parallelize(range(5)) rdd = rdd.map(lambda x: x + 1) rdd.collect() Here you have distributed collection and lambda function to send to all workers. Lambda ...


10

Take a look at the polyglot library. It has polarity lexicons for 136 languages, including Russian. The scale of the words’ polarity consisted of three degrees: +1 for positive words, and -1 for negatives words. Neutral words will have a score of 0. You can use it like this: >>> from polyglot.text import Text as T >>> text = T("это ...


10

1- The number of features: In terms of neural network model it represents the number of neurons in the projection(hidden) layer. As the projection layer is built upon distributional hypothesis, numerical vector for each word signifies it's relation with its context words. These features are learnt by the neural network as this is unsupervised method. Each ...


9

Parts of Speech (POS) This is what it is called when you label each of the words (often called tokens) of a sentence or many sentences. Usually they are labelled with grammatical descriptions, such as Noun, Adjective, Adverb. They can often get quite specific, also distinguishing e.g. between types of nouns (proper nouns etc). You can then use these ...


8

You can use the SAR14 dataset of 234K IMDb movie reviews. The construction of the SAR14 dataset is detailed in the paper "Sentiment Classification on Polarity Reviews: An Empirical Study Using Rating-based Features".


6

If you are seeking a working solution, I know of an API that supports many languages, including Russian: indico.io Text Analysis sentiment() >>> import indicoio >>> indicoio.config.api_key = YOUR_API_KEY >>> indicoio.sentiment(u"Это круто, убивает! Хочу.", language='ru') 0.6978093435482927 >>> indicoio.sentiment(u"Ты кто ...


6

Let me suggest three simple options: average the vectors (component-wise), i.e., compute the word embedding vector for each word in the text, and average them. (as suggested by others). take the (component-wise) maximum of the vectors. (max, instead of average) take the (component-wise) minimum of the vectors. (min, instead of average) Each of these ...


6

First of all good job done in processing the data and coming up with your base model. I would suggest few things that you can try: Improve your model my adding bigrams and tri-grams as features. Try doing some topic modelling like latent Dirichlet allocation or Probabilistic latent Semantic Analysis for the corpus using a specified number of topics - say ...


6

The embedding matrix which used in the initialization of the Embedding layer is highly trained on a large corpus of text. The training and the data are so huge that the embedding has learnt a type of association between words. A pretrained embedding like Word2Vec will produce vectors for words like school and homework which are similar to each other in ...


5

Consider the following two sentences: My awesome girlfriend bought me a delicious popsicle at the store. 0--1-------0----------0------0--0-1---------0--------0--0---0-----:2:11 My awesome girlfriend, Joyce, drove to the grocery store to buy me a delicious Dole popsicle. 0---1------0-----------0------0-----0--0---0-------0-----0--0---0--0-1---------0----0---...


5

A couple of important points: Sentiment analysis is not an exact science. Two people, reading the same text in different contexts will come to different conclusions about sentiment, especially on borderline cases. Perhaps text has complex grammar, or has a metaphor or simile in it where it helps to understand what is actually being compared. The ground ...


5

The final range of emotion is completely arbitrary. No matter the interval [a, b], you can adjust the emotions to fit inside. [-100, 100] is perfectly reasonable and is common. An example of use is from GDELT, which provides this interval for average tone of news documents. Asking if equally distancing the emotions is statistically correct does not make ...


5

You can add a second Input layer to your architecture. Look at this link Keras Guide Multiple Inputs for a more detailed explanation. The output of the embedding layer can be combined with the secondary input and passed to the next layer. auxiliary_input = Input(shape=(5,), name='aux_input') x = keras.layers.concatenate([lstm_out, auxiliary_input])


5

Feature Extraction is an important step when dealing with natural languages because the text you've collected isn't in a form understandable by a computer. If you have a tweet that goes something like I do not like the views of @Candidate1 on #Topic1. Too conservative!! I can't stand it! then we can't just feed this into a learning algorithm. We need to ...


5

With your three labels: positive, neutral or negative - it seems you are talking more about sentiment analysis. This answer the question: what are the emotions of the person who wrote this piece of text? Semantic analysis is a larger term, meaning to analyse the meaning contained within text, not just the sentiment. It looks for relationships among the ...


4

You can classify a few of the tweets yourself, and compare afterwards which of the two algorithmic results is closer to your classification. Without more information we cannot tell what these algorithms were doing. It may well be that they were just using different thresholds internally: Algo 1 decided that everything > 60% threshold is "positive", all < ...


4

The NLTK book is by far the best tutorial on basic NLP I have seen(in Python). The Coursera course on NLP is also fairly good. It takes off from the basics and takes the student to a novice level.


4

Your formula is correct for one $w_i$, but if you want to classify a document, you need to compute $P(c | w_1,\ldots,w_N)$. Then you have $$P(c | w_1,\ldots,w_N) = \frac{P(c)\cdot P(w_1,\ldots,w_N|c)}{P(w_1,\ldots,w_N)} = \frac{P(c) \cdot \prod_{i=1}^N P(w_i|c)}{P(w_1,\ldots,w_N)} \neq \prod_{i=1}^NP(c|w_i)$$ where the second equation holds because of the ...


4

Collecting your data From the comments you state that you wish to classify comments into a label (1-poor, 2-fair, 3-ok, 4-good, 5-very good). Thus you will be training a model that maps a set of words (the paragraph) to a number which represents the rating. I will assume that you also have labels for some of the comments which we will call your training set....


3

Two very different suggestions here to avoid averging the vectors: Use Word Mover's Distance (https://github.com/mkusner/wmd) to compute distance between the tweets (not sure how well it would work on short texts like tweets, I still need to try that myself...) Cluster the word vectors themselves (using e.g. kmeans), then for each tweet create a vector with ...


3

There are many datasets available. Multi-Domain Sentiment Dataset Twitter sentiment UCI Sentiment Analysis Dataset Large Movie Review Dataset


3

The quick (and not very satisfying) answer is "it depends" -- specifically it depends upon what your underlying conceptual model of human emotion is and how it manifests in verbal/written behavior. What is your characterization of neutrality in relation to positive and negative valence? Can documents be put on some sort of quantitative scale with neutral ...


3

I was thinking if Apriori might be more suited for your purpose. For your consideration: 1) Tokenize the training sentences into bag of words: Review Sentence | Upside | Calm | Swimming | Cat 2) Tag the correct consequents for bag of words. 3) Apriori should produce 1 rule for ambiance and 1 rule for Entertainment. Hope this helps.


3

The Indico.io API supports Spanish (and Chinese (Mandarin), Japanese, Italian, French, Russian, Arabic, German, English). eg in Python: >>> import indicoio >>> indicoio.config.api_key = <YOUR_API_KEY> >>> indicoio.sentiment("¡Jamás voy a usar esta maldita aplicación! No funciona para nada.") 0.02919392219306888 >>>...


3

The Stanford NLP course on Coursera covers Sentiment Analysis in week 3: - What is Sentiment Analysis? - Sentiment Analysis: A baseline algorithm - Sentiment Lexicons - Learning Sentiment Lexicons - Other Sentiment Tasks For coding tutorials see: Stream Hacker's NLP tutorials Basic Sentiment Analysis with Python Andy Bromberg's Sentiment Analysis ...


3

The easiest way is to average the word- embeddings. This works quite well. Another thing you can try is to represent each document as a bag of words - i.e. - to have a vector in the size of your vocabulary, where each element in the vector represents the number of times a certain word had been mentioned in your document (for example, the first element in ...


3

Only full sentences are used for testing and validation, though sentences and phrases are used for training. See Yoon Kim 2014, Convolutional Neural Networks for Sentence Classification for an overview of various text classification experiments. Here is code that creates training, dev, and test .CSV files from the various text files in the dataset download. ...


3

Parts of Speech explains how a word is used in a sentence, i.e whether it is a verb, noun, adjective and so on. In text processing, those POS (or word classes) are usually represented as their abbreviation and we call it tag. For example if we use nltk, it uses The Penn Treebank tagset as a default. https://www.ling.upenn.edu/courses/Fall_2003/ling001/...


3

CLS stands for classification and its there to represent sentence-level classification. In short in order to make pooling scheme of BERT work this tag was introduced. I suggest reading up on this blog where this is also covered in detail.


Only top voted, non community-wiki answers of a minimum length are eligible