Hot answers tagged

15

The closest would be like Jan has mentioned inhis answer, the Levenstein's distance (also popularly called the edit distance). In information theory and computer science, the Levenshtein distance is a string metric for measuring the difference between two sequences. Informally, the Levenshtein distance between two words is the minimum number of ...


13

Word2vec does not capture similarity based on antonyms and synonyms. Word2vec would give a higher similarity if the two words have the similar context. Eg The weather in California was _____ . The blank could be filled by both hot and cold hence the similarity would be higher. This concept is called Paradigmatic relations. If you are interested to capture ...


9

Apart from very good responses here, you may try SequenceMatcher in difflib python library. https://docs.python.org/2/library/difflib.html import difflib a = 'Thanks for calling America Expansion' b = 'Thanks for calling American Express' seq = difflib.SequenceMatcher(None,a,b) d = seq.ratio()*100 print(d) ### OUTPUT: 87.323943 Now Consider the below ...


7

In Text Analytic Tools for Semantic Similarity, they developed a algorithm in order to find the similarity between 2 sentences. But if you read closely, they find the similarity of the word in a matrix and sum together to find out the similarity between sentences. So, it might be a shot to check word similarity. Also in SimLex-999: Evaluating Semantic ...


6

If your dictionary is not too big a common approach is to take the Levenshtein distance, which basically counts how many changes you have to make to get from one word to another. Changes include changing a character, removing a character or adding a character. An example from Wikipedia: lev(kitten, sitting) = 3 k itten -> s itten sitt e n -> sitt i n ...


5

Look up taxonomy/ontology construction/induction. Relevant papers: Automatic Taxonomy Construction from Keywords via Scalable Bayesian Rose Trees Topic Models for Taxonomies OntoLearn Reloaded. A Graph-Based Algorithm for Taxonomy Induction Ontology Population and Enrichment: State of the Art Probabilistic Topic Models for Learning Terminological Ontologies


5

It all depends on your definition of what a common word is in your domain. You are using an NLTK corpus which likely doesn't fit your domain very well. Either you have a corpus containing the domain you want and you do a simple lookup. Or you don't know in advance and you need to compute these common words from your documents (your short phrases). In that ...


4

Except for the OCR part, the right bundle would be pandas and sklearn. You can check this ipython notebook which uses TfidfVectorizer and SVC Classifier. This classifier can make one-vs-one or one-vs-the-rest multiclass predictions, and if you use the predict_proba method instead of predict, you would have the confidence level of each category. If you're ...


4

Assuming you are doing supervized learning to train a model that when deployed will take text as input and output a label (e.g., topic) or class probability, then what you probably want to do is balanced, stratified sampling. Assuming sufficient labelled data, ensure that your final training set has a balanced number of text examples for each class/label. ...


4

To build off Mashimo's answer, one straightforward approach for topic modeling is "Latent Dirichlet Allocation" (LDA). The basic idea behind LDA is explained in this really good tutorial. Essentially, documents are assumed to be composed of mixtures of topics, which are in turn composed of mixtures of words. If we knew the topic and document distributions, ...


4

I am staying quite generic since you asked for enlightenment, just mentioning some possible directions that you can explore. You have basically two possibilities: Classification of the text (Supervised learning). Supervised means that you need first to externally apply labels (for example manually by humans) to examples of texts (labels could be "politics"...


4

Try something like this: import matplotlib.pyplot as plt plt.figure(figsize=(30, 20)) # the size you want # your code goes here


4

Big problem and very good question! I used spacy in the past, which has a German module. I guess stemming is not supported, but lemmatization. Looking at the output below, I don't think that spacy will solve your problem to be honest. However, I just wanted to let you know about this option. Spacy Lemmatization: #pip install spacy #python -m spacy ...


3

It is unclear if you are requesting AUC of ROC or Precision-Recall curve. However, instead of storing the indices of examples in sets, you can store the labels in lists and use sklearn's auc function after running precision_recall_curve or roc_curve: from sklearn.metrics import precision_recall_curve from sklearn.metrics import roc_curve from sklearn....


3

Hey, Here's how i would solve this. The problem with regex is that you don't have any index so I find it better to convert your inputs into lists of single words/tags by splitting on spaces. Then it is easier to find the pattern in the tag list and get the matching words. words = [word for key in key_list for word in key.split()] tags = [tag for value ...


3

Word2vec is a good starting point for most scenarios. It does capture semantics by way of prediction using CBOW method. It allows translations (as most repeated example I can put here again), V(King) - V(Queen) ~~ V(men) - V(women) and so on. So what is the problem? The issue lies in word sense ambiguity. Whenever the word itself has two different meaning ...


3

An old and well-known technique for comparison is the Soundex algorithm. The idea is to compare not the words themselves but approximations of how they are pronounced. To what extent this actually improves the quality of the results I don't know. However it feels a bit strange to apply something like Soundex to results from a speech-to-text recognition ...


3

Tag mapping according to nltk source 'CC': 'Coordinating conjunction', 'PRP$': 'Possessive pronoun', 'CD': 'Cardinal number', 'RB': 'Adverb', 'DT': 'Determiner', 'RBR': 'Adverb, comparative', 'EX': 'Existential there', 'RBS': 'Adverb, superlative', 'FW': 'Foreign word', 'RP': '...


3

Check out chapter 6 section 2.2 of the NLTK book. EDIT: since apparently the community wants me to copy/paste stuff, here ya go: 2.2 Identifying Dialogue Act Types When processing dialogue, it can be useful to think of utterances as a type of action performed by the speaker. This interpretation is most straightforward for performative statements such as ...


3

No one answered the question, then I will answer it myself. Thanks to @user12075 for the link. I didn't find it when I was googling it. https://stackoverflow.com/questions/15388831/what-are-all-possible-pos-tags-of-nltk From the above link, I know that nltk uses The Penn Treebank's POS tags. nltk.help.upenn_tagset() will give you the list.


3

One way is to loop through a list of sentences. Process each one sentence separately and collect the results Brian is totally right about the solution but I think that's actually the part that is missing from his answer :) Brian's code assumes that the sentences have already been segmented, which was Ahmad's original question as far as I can tell. With ...


3

There are a couple of items that could be improved in your code: nltk.corpus.stopwords is a nltk.corpus.util.LazyCorpusLoader. You might want stopwords.words('english'), a list of stop English words. It can cause bugs to update a variable as you iterate through it, for example sentance in your code. In your code preprocessed_reviews is not being updated. ...


3

Some common approaches to this problem are: Keep only the n- most common words in a corpus (automatically done in scikit-learn: https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer). Keep all words, but downweight uninformative words using a transformation ...


2

NLTK has a built-in NER model that would extract potential Organizations from text, you can read about it here (and see examples) NLTK book (look for section "5 Named Entity Recognition"). However, if your input text has organizations in a very specific context that wasn't seen by NLTK NER model, performance might be quite low. In that case you should be ...


2

your grammar is correct! grammar = """MEDIA: {<DT>?<JJ>*<NN.*>+} RELATION: {<V.*>} {<DT>?<JJ>*<NN.*>+} ENTITY: {<NN.*>}""" by specifying RELATION: {<V.*>} {<DT>?<JJ>*<NN.*>+} you are indicating that there are two ways to generate ...


2

Given that exact case, I would assume that you are getting negative decision due to names, mentioned in the review (in your training dataset actors were more often met in negative reviews). You should, probably, remove all non-relevant words from reviews, and that includes not only stop words, but all person names (since they are less of a sentiment marker ...


2

You can have a look into this repository, it was recently announced in hacker news. I personally don't have experience using it, but the benchmarks look interesting: prose is a natural language processing library (English only, at the moment) in pure Go. It supports tokenization, segmentation, part-of-speech tagging, and named-entity extraction.


2

No. Not yet There is no single package in Golang, which acts as versatile as nltk for NLP. However, there are several packages which aim to do it. Here is a compiled list of such packages: https://github.com/gopherds/resources/blob/master/tooling/README.md#nlp


2

It looks like you installed it only on the driver/gateway and not on the nodes/workers itself. The test you ran in the shell is running it locally, once you map a function via your SparkContext it gets distributed to the workers which don't have NLTK installed.


2

The multiword tokenizer 'nltk.tokenize.mwe' basically merges a string already divided into tokens, based on a lexicon, from what I understood from the API documentation. One thing you can do is tokenize and tag all words with it's associated part-of-speech (PoS) tag, and then define regular expressions based on the PoS-tags to extract interesting key-...


Only top voted, non community-wiki answers of a minimum length are eligible