Hot answers tagged

20 votes
Accepted

Similarity between two words

The closest would be like Jan has mentioned inhis answer, the Levenstein's distance (also popularly called the edit distance). In information theory and computer science, the Levenshtein distance ...
user avatar
  • 8,036
18 votes

How can I get a measure of the semantic similarity of words?

Word2vec does not capture similarity based on antonyms and synonyms. Word2vec would give a higher similarity if the two words have the similar context. Eg The weather in California was _____ . The ...
user avatar
13 votes

Similarity between two words

Apart from very good responses here, you may try SequenceMatcher in difflib python library. https://docs.python.org/2/library/difflib.html ...
user avatar
  • 271
10 votes

How can I get a measure of the semantic similarity of words?

In Text Analytic Tools for Semantic Similarity, they developed a algorithm in order to find the similarity between 2 sentences. But if you read closely, they find the similarity of the word in a ...
user avatar
  • 2,296
6 votes

Similarity between two words

If your dictionary is not too big a common approach is to take the Levenshtein distance, which basically counts how many changes you have to make to get from one word to another. Changes include ...
user avatar
5 votes
Accepted

Inferring Relational Hierarchies of Words

Look up taxonomy/ontology construction/induction. Relevant papers: Automatic Taxonomy Construction from Keywords via Scalable Bayesian Rose Trees Topic Models for Taxonomies OntoLearn Reloaded. A ...
user avatar
  • 10.4k
5 votes

How to change plot size in nltk.plot()

Try something like this: import matplotlib.pyplot as plt plt.figure(figsize=(30, 20)) # the size you want # your code goes here
user avatar
  • 4,046
5 votes

Is there a good German Stemmer?

Big problem and very good question! I used spacy in the past, which has a German module. I guess stemming is not supported, but lemmatization. Looking at the ...
user avatar
  • 6,922
5 votes
Accepted

Detect if word is «common English» word or slang word

It all depends on your definition of what a common word is in your domain. You are using an NLTK corpus which likely doesn't fit your domain very well. Either you have a corpus containing the domain ...
user avatar
4 votes

Similarity between two words

An old and well-known technique for comparison is the Soundex algorithm. The idea is to compare not the words themselves but approximations of how they are pronounced. To what extent this actually ...
user avatar
4 votes
Accepted

Document Categorization Problem

Except for the OCR part, the right bundle would be pandas and sklearn. You can check this ipython notebook which uses ...
user avatar
  • 156
4 votes
Accepted

Creating training data

Assuming you are doing supervized learning to train a model that when deployed will take text as input and output a label (e.g., topic) or class probability, then what you probably want to do is ...
user avatar
4 votes

NLTK Sklearn Genism Text to Topic

I am staying quite generic since you asked for enlightenment, just mentioning some possible directions that you can explore. You have basically two possibilities: Classification of the text (...
user avatar
  • 41
4 votes

NLTK Sklearn Genism Text to Topic

To build off Mashimo's answer, one straightforward approach for topic modeling is "Latent Dirichlet Allocation" (LDA). The basic idea behind LDA is explained in this really good tutorial. Essentially, ...
user avatar
  • 331
4 votes

How to extract Question/s from document with NLTK?

Check out chapter 6 section 2.2 of the NLTK book. EDIT: since apparently the community wants me to copy/paste stuff, here ya go: 2.2 Identifying Dialogue Act Types When processing dialogue, it ...
user avatar
  • 3,028
4 votes

Where to know the list of NLTK tagset?

No one answered the question, then I will answer it myself. Thanks to @user12075 for the link. I didn't find it when I was googling it. https://stackoverflow.com/questions/15388831/what-are-all-...
user avatar
4 votes
Accepted

nltk's stopwords returns "TypeError: argument of type 'LazyCorpusLoader' is not iterable"

There are a couple of items that could be improved in your code: nltk.corpus.stopwords is a nltk.corpus.util.LazyCorpusLoader. ...
user avatar
4 votes
Accepted

Is there any package in python that can identify similarity between alphanumeric alias names of a parameter?

If you know there are only specific variants, you can obviously make a look-up table yourself (i.e. a Python dictionary). Otherwise you could try using a fuzzy matching library, like fuzzywuzzy. This ...
user avatar
  • 14k
4 votes

Is there an NLP corpus that contains common medical terms?

Welcome to the biomedical domain, one of the few domains in NLP where there are too many resources to choose from :) Data resources: Medline is a database corpus of 30 millions abstracts. Each ...
user avatar
  • 22.2k
3 votes
Accepted

I need to measure Performance : AUC for this code of NLTK and skLearn

It is unclear if you are requesting AUC of ROC or Precision-Recall curve. However, instead of storing the indices of examples in sets, you can store the labels in lists and use sklearn's ...
user avatar
  • 1,153
3 votes

Filter phrases based on correspondent POS tags

Hey, Here's how i would solve this. The problem with regex is that you don't have any index so I find it better to convert your inputs into lists of single words/tags by splitting on spaces. Then it ...
user avatar
  • 31
3 votes

How can I get a measure of the semantic similarity of words?

Word2vec is a good starting point for most scenarios. It does capture semantics by way of prediction using CBOW method. It allows translations (as most repeated example I can put here again), V(King) -...
user avatar
3 votes
Accepted

What is the tag mapping for entity recognition in nltk?

Tag mapping according to nltk source ...
user avatar
3 votes

Accuracy of word and sent tokenize versus custom tokenizers in nltk

Why would we want a custom tokenizer? Segementation is a very large topic, and as thus there is no perfect Natural Language Tokenizer. Any toolkit needs to be flexible, and the ability to change ...
user avatar
  • 1,753
3 votes

Why are Chunking and IOB tags necessary?

BIO(L) tagging is important (but as you correctly noted, not necessary) part of a NER pipeline. Main idea behind such split is to facilitate learning in following manner. Take English as an example, ...
user avatar
3 votes

sentiment analysis nltk python

What you're doing right now is a traditional classification using supervised learning. This is a great method for predicting outcomes, but I suspect there are much better ways to complete this ...
user avatar
3 votes

How to use bigrams for a text of sentences?

One way is to loop through a list of sentences. Process each one sentence separately and collect the results Brian is totally right about the solution but I think that's actually the part that is ...
user avatar
3 votes

Detect if word is «common English» word or slang word

Some common approaches to this problem are: Keep only the n- most common words in a corpus (automatically done in scikit-learn: https://scikit-learn.org/stable/modules/generated/sklearn....
user avatar
  • 119
2 votes
Accepted

Python: validating the existence of NLTK data with database search

NLTK has a built-in NER model that would extract potential Organizations from text, you can read about it here (and see examples) NLTK book (look for section "5 Named Entity Recognition"). However, ...
user avatar
2 votes
Accepted

Complex Chunking with NLTK

your grammar is correct! ...
user avatar
  • 368

Only top scored, non community-wiki answers of a minimum length are eligible