11

Assuming fairly reasonable data normalization, the expectation of the weights should be zero or close to it. It might be reasonable, then, to set all of the initial weights to zero because a positive initial weight will have further to go if it should actually be a negative weight and visa versa. This, however, does not work. If all of the weights are the ...


10

I would say that lemmatization is generally the preferred way of reducing related words to a common base. This Quora question is a good resource on the subject: Is it advisable to choose lemmatization over stemming in NLP? The top answer quotes another good resource that motivates why lemmatization is usually better, Stemming and lemmatization, from ...


5

With your three labels: positive, neutral or negative - it seems you are talking more about sentiment analysis. This answer the question: what are the emotions of the person who wrote this piece of text? Semantic analysis is a larger term, meaning to analyse the meaning contained within text, not just the sentiment. It looks for relationships among the ...


5

A couple of important points: Sentiment analysis is not an exact science. Two people, reading the same text in different contexts will come to different conclusions about sentiment, especially on borderline cases. Perhaps text has complex grammar, or has a metaphor or simile in it where it helps to understand what is actually being compared. The ground ...


4

You can classify a few of the tweets yourself, and compare afterwards which of the two algorithmic results is closer to your classification. Without more information we cannot tell what these algorithms were doing. It may well be that they were just using different thresholds internally: Algo 1 decided that everything > 60% threshold is "positive", all < ...


3

The original paper mentions two corpora: CoNLL 2003 (apparently here now) and the "CMU Seminar Announcements Task". However according to the page linked in the question the actual NER was trained on a larger combination of corpora: Our big English NER models were trained on a mixture of CoNLL, MUC-6, MUC-7 and ACE named entity corpora, and as a result the ...


3

Not sure what the video said, but $T$ should not be the vocabulary size, but the training corpus size (number of all words). For example, if your training corpus is deep learning is popular . i love deep learning . i want to learn more about it. Then when you sume up over $T$, you will sum up all the word pairs in the corpus including duplicates. The ...


3

If you set it as 0, they will all have the same error so backprop will make them all equal; therefore, you should have random initialisation. Why around 0? I think this post may answer it well: https://stats.stackexchange.com/questions/47590/what-are-good-initial-weights-in-a-neural-network


3

It depends on how similarity is defined. If similarity is defined as human-defined semantics, then a synset (i.e., synonym set) is most appropriate. If similarity is defined as frequent co-occurrence, then word embeddings are most appropriate. Even within semantic similarity, there are many approaches beyond synsets. One advantage of word embeddings over ...


3

Yes, there is a french model free and ready to use via the spaCy package! Here are the small amd medium sized models, that should be ready to go. Here is the basic summary of the dataset, shown at the spaCy website:


3

The OP asks two different questions: (1) how to extract key words and (2) how to assign keywords a sentiment class (pos/neg/neu). I will address the keyword identification piece in this answer as many others have discussed how to do sentiment analysis (e.g., this post). The approach I would suggest is a key keyword approach advocated by Mike Scott (author ...


2

Name parsing does not appear to built-in to Stanford CoreNLP. . One option is writing a series of Regular Expression using Stanford TokensRegex to parse and label name tokens. Another option is using a third party package, such as nameparser in Python.


2

If sensitivity to case is breaking your models you have two options: Train or find a new model that's case-insensitive. This is probably the easiest thing to do. The Stanford parser has one. Train a model to correct the case of your input, this is sometimes called truecasing. The Stanford Parser has this functionality too.


2

Named Entity Recognition is technique which can be used here. Location is one of the 3 most studied classes (with Person and Organization). Stanford NLP has an open source Java implementation that is extremely powerful. For Example let say sentence is "i will be going to Sweden from Boston." Now here you can use regular expression to detect these ...


2

@imoutidi, I also encountered the same deprecation warning. After digging around a bit, it looks like the new/replacement package can be imported with the following: from nltk.tag.stanford import CoreNLPNERTagger However, when trying to run the tag() method I end up getting an unexpected HTTP connection refused error. I haven't figured out if this is an ...


1

IT is very similiar to PMI, here you just expand it to the whole dictionary matrix (matrix representation of the whole vocabulary), normalize it by subtracting quantitive representation of the sum of words found in row i column j and than standardize. (Like when using sklearn Standardize(), similiar atleast) Intuition? Well why is tf-idf working (generally ...


1

The purpose of smoothing is to prevent a language model from assigning zero probability to unseen events. That is needed because in some cases, words can appear in the same context, but they didn't in your train set. Smoothing is a quite rough trick to make your model more generalizable and realistic. You can also see it as a tool to prevent overfitting.


1

Have a look at this to see how a Knowledge Base can be built. I would say the best way is to build your own Knowledge Base based on your corpus if you have enough data. The idea (simplified) for starting is to have patterns such as "NAME is POSITION" by seeing some of data. Through this you find many names and positions. Then you extract new patterns from ...


1

I do not know of anything called StanfordCoreNLPParser. The stanfordcorenlp package has StanfordCoreNLP: from stanfordcorenlp import StanfordCoreNLP


1

Looking at your code, I dont believe there is any mistake in it. The Stanford NER tagger is modelled by a Conditional Random Field Classifier and is hence bound to make mistakes. If you are able to check for a large number of samples and calculate an accuracy metric on the basis of that you might get a good idea if the pre-trained model is useful for you. ...


1

I think these are the methods that you can try out (Please feel free to add more to this list): Highly precise with a little low recall is to use a dictionary with almost all possibilities (manual effort, but must be worth it.). Using Word2Vec. Mikolov has already trained text data and created word vectors. Using this vector space, you can figure out which ...


1

It's not a straightforward question to answer as it is hard to compare the quality of two word2vec models with a meaningful metric. You could, of course, use the loss function, but that won't give much. Another approach is more heuristic: take for example the frequency of each word, and remove those words that are repeated less than N times, where you can ...


1

If you don t know the number of clusters, i encourage you to look at those density based algorithm : Mean Shift, DBSCAN, OPTICS. They don t presume of the cluster number and are able to find random shape clusters.


1

A minimum cluster size will not generally be satisfiable in hierarchical clustering. Instead, you have to expect many clusters with just a single point. ELKI has some fairly interesting techniques to cut a dendrogram. Check the clustering.hierarchical.extraction (or so) package. If I remember correctly, some allow you to set a minimum size (but there will ...


1

Open Calais is a free-to-use tool for entity recognition and relationship mapping. It's from Thompson-Reuters so may not be wholly suitable for technical language but worth a try. Has python bindings


1

I'm not an expert here, so here's my (brute force?) method. SeatGeek has open-sourced a python library called fuzzywuzzy which is great at text matching. It has a function called token_set_ratio that compares two multi-word strings and scores their distance. It can consider just the intersection of individual words and score only that intersection. eg "...


1

This should have been a comment but i don't have the reputation. There are multiple ways: Simplest Approach - Use textblob to find polarity and add the polarity of all sentences. If the overall polarity of tweet is greater than 0 , then it's positive and if less than zero , you can label it as negative Use of lexicons- One can use MQPA lexicon , to find ...


1

One easy solution is to prepare a emotion dictionary first. Such dictionary can be found online easily such as http://www.psychpage.com/learning/library/assess/feelings.html. A simple workflow will be as follow: For each tweet, tokenize the tweet into a list of vocabularies For each list of vocabularies, count the (positive, neutral, negative) words ...


1

I'm sorry I'm not a Java user and I never worked with StanfordNLP. But I do know that the gini impurity criterion with and without decision tree were successfully applied for text classification. Moreover these tools have the ability to let you easily understand which features ( i.e. words in your case) contribute to the decision. I know that ...


1

You'll need labeled data. It’s best if you have your own, but if you don’t, make sure you take your labeled data from the "Twitter" domain. This is important because tweets have a very specific style of writing, a specific lexicon, and use many emoticons. To help, here’s a database of 5K manually labeled tweets from Nick Sanders: http://www.sananalytics....


Only top voted, non community-wiki answers of a minimum length are eligible